Validation error not enough nodes you need at least three nodes to complete this process

How to troubleshoot Elasticsearch/OpenSearch log "not enough master nodes discovered during pinging" a detailed guide including background on ES concepts: discovery and master

Overview

The process known as discovery occurs when an Elasticsearch node starts, restarts or loses contact with the master node for any reason. In those cases, the node needs to contact other nodes in the cluster to find any existing master node or initiate the election of a new master node. 

How it works

Upon startup, each node looks for other nodes, firstly by contacting the IP addresses of eligible master nodes held in the previous cluster state.  If they are not available, it will look for nodes based upon the seed host provider mechanisms available.

Seed host providers may be defined in 3 ways: list based, file based or plugin based. All of these methods provide a list of IP addresses or hostnames which the node should contact in order to obtain a list of master eligible nodes. The node will contact all of these addresses in turn, until either an active master is found, or failing that, until sufficient nodes can be found to elect a new master node.

Examples

The simplest form is to define a list of seed host providers in elasticsearch.yml:

discovery.seed_hosts:
   - 192.168.1.10:9300
   - 192.168.1.11 
   - seeds.mydomain.com

An alternative way is to refer to a file using the following setting:

discovery.seed_providers: file

The file MUST be placed in the following filepath: $ES_PATH_CONF/unicast_hosts.txt

10.10.10.5
10.10.10.6:9305
10.10.10.5:10005
# an IPv6 address
[2001:0db8:85a3:0000:0000:8a2e:0370:7334]:9301

Note that the use of a port is optional. If not used, then the default port range of 9300-9400 will be used.

If you use AWS or GCS then you can install and use a plugin to obtain a list of seed hosts from an API.  A plugin also exists for Azure but is deprecated since version 5.

AWS plugin

A typical configuration could be as follows:

discovery.seed_providers: ec2
discovery.ec2.tag.role: master
discovery.ec2.tag.environment: dev
discovery.ec2.endpoint: ec2.us-east-1.amazonaws.com
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone

The above configuration would look for all nodes with a tag called “environment” set to “dev” and a tag called “role” set to “master”, in the AWS zone us-east-1. The last two lines set up cluster routing allocation awareness based upon aws availability zones. (Not necessary, but nice to have).

GCE plugin

A typical configuration could be as follows:

discovery.seed_providers: gce
cloud.gce.project_id: <your-google-project-id>
cloud.gce.zone: <your-zone>
discovery.gce.tags: <my-tag-name>

The above configuration would look for all virtual machines inside your project, zone and with a tag set to the tag name you provide.

Notes and good things to know

Cluster formation depends on correct setup of the network.host settings in elasticsearch.yml.  Make sure that the nodes can reach each other across the network using their IP addresses / hostname, and are not getting blocked due to firewall settings on the ports required.

Источник

Chart version:
7.1.1
Kubernetes version:
1.13.0
Kubernetes provider: E.g. GKE (Google Kubernetes Engine)
Kubespray, 3 master, 3 Worker Node
Helm Version:
2.13.0
helm get release output

e.g. helm get elasticsearch (replace elasticsearch with the name of your helm release)

Describe the bug:

Steps to reproduce:

  1. Clone the latest from repo
  2. Run «helm install —name elasticsearch . —version 7.1.1»

Expected behavior:
Cluster are not discovered or joining properly
Provide logs and/or server output (if relevant):

	{"type": "server", "timestamp": "2019-06-18T16:57:12,863+0000", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "setting initial configuration to VotingConfiguration{OHs2L7FRTFS_o_wPoWEgwA,{bootstrap-placeholder}-elasticsearch-master-2,XJkU43IQSh2C7cqf8tvwwA}"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:41,459+0000", "level": "WARN", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "timed out while waiting for initial discovery state - timeout: 30s"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:41,464+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [OHs2L7FRTFS_o_wPoWEgwA, XJkU43IQSh2C7cqf8tvwwA], have discovered [{elasticsearch-master-1}{XJkU43IQSh2C7cqf8tvwwA}{x2DLZR_sROq83l_EAwPcuQ}{10.233.64.62}{10.233.64.62:9300}{ml.machine_memory=4294967296, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{sKE7xbPKQMydzd1BmO-1zw}{V3gnvyR9SzGCenr3cUpVaA}{10.233.108.59}{10.233.108.59:9300}{ml.machine_memory=4294967296, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [10.233.108.59:9300, 10.233.64.62:9300] from hosts providers and [{elasticsearch-master-0}{OHs2L7FRTFS_o_wPoWEgwA}{l8PW0iCmS82lr96DK6baOg}{10.233.92.41}{10.233.92.41:9300}{ml.machine_memory=4294967296, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:41,468+0000", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "publish_address {10.233.92.41:9200}, bound_addresses {0.0.0.0:9200}"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:41,469+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "started"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:47,462+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "no known master node, scheduling a retry"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:48,464+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "timed out while retrying [cluster:monitor/health] after failure (timeout [1s])"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:48,466+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}" , 
	"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
	"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.1.jar:7.1.1]",
	"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
	"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
	"at java.lang.Thread.run(Thread.java:835) [?:?]"] }
	{"type": "server", "timestamp": "2019-06-18T16:57:55,861+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [OHs2L7FRTFS_o_wPoWEgwA, XJkU43IQSh2C7cqf8tvwwA], have discovered [{elasticsearch-master-1}{XJkU43IQSh2C7cqf8tvwwA}{x2DLZR_sROq83l_EAwPcuQ}{10.233.64.62}{10.233.64.62:9300}{ml.machine_memory=4294967296, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{sKE7xbPKQMydzd1BmO-1zw}{V3gnvyR9SzGCenr3cUpVaA}{10.233.108.59}{10.233.108.59:9300}{ml.machine_memory=4294967296, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [10.233.108.59:9300, 10.233.64.62:9300] from hosts providers and [{elasticsearch-master-0}{OHs2L7FRTFS_o_wPoWEgwA}{l8PW0iCmS82lr96DK6baOg}{10.233.92.41}{10.233.92.41:9300}{ml.machine_memory=4294967296, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 1, last-accepted version 0 in term 0"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:57,409+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "no known master node, scheduling a retry"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:58,410+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "timed out while retrying [cluster:monitor/health] after failure (timeout [1s])"  }
	{"type": "server", "timestamp": "2019-06-18T16:57:58,411+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}" , 
	"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
	"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.1.jar:7.1.1]",
	"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
	"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
	"at java.lang.Thread.run(Thread.java:835) [?:?]"] }
	{"type": "server", "timestamp": "2019-06-18T16:58:07,409+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "no known master node, scheduling a retry"  }
	{"type": "server", "timestamp": "2019-06-18T16:58:08,410+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "timed out while retrying [cluster:monitor/health] after failure (timeout [1s])"  }
	{"type": "server", "timestamp": "2019-06-18T16:58:08,411+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}" , 
	"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
	"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.1.jar:7.1.1]",
	"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
	"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
	"at java.lang.Thread.run(Thread.java:835) [?:?]"] }
	{"type": "server", "timestamp": "2019-06-18T16:58:10,304+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [OHs2L7FRTFS_o_wPoWEgwA, XJkU43IQSh2C7cqf8tvwwA], have discovered [{elasticsearch-master-1}{XJkU43IQSh2C7cqf8tvwwA}{x2DLZR_sROq83l_EAwPcuQ}{10.233.64.62}{10.233.64.62:9300}{ml.machine_memory=4294967296, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{sKE7xbPKQMydzd1BmO-1zw}{V3gnvyR9SzGCenr3cUpVaA}{10.233.108.59}{10.233.108.59:9300}{ml.machine_memory=4294967296, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [10.233.108.59:9300, 10.233.64.62:9300] from hosts providers and [{elasticsearch-master-0}{OHs2L7FRTFS_o_wPoWEgwA}{l8PW0iCmS82lr96DK6baOg}{10.233.92.41}{10.233.92.41:9300}{ml.machine_memory=4294967296, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 2, last-accepted version 0 in term 0"  }
	{"type": "server", "timestamp": "2019-06-18T16:58:10,305+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "failed to join {elasticsearch-master-0}{OHs2L7FRTFS_o_wPoWEgwA}{l8PW0iCmS82lr96DK6baOg}{10.233.92.41}{10.233.92.41:9300}{ml.machine_memory=4294967296, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-0}{OHs2L7FRTFS_o_wPoWEgwA}{l8PW0iCmS82lr96DK6baOg}{10.233.92.41}{10.233.92.41:9300}{ml.machine_memory=4294967296, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={elasticsearch-master-0}{OHs2L7FRTFS_o_wPoWEgwA}{l8PW0iCmS82lr96DK6baOg}{10.233.92.41}{10.233.92.41:9300}{ml.machine_memory=4294967296, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{OHs2L7FRTFS_o_wPoWEgwA}{l8PW0iCmS82lr96DK6baOg}{10.233.92.41}{10.233.92.41:9300}{ml.machine_memory=4294967296, xpack.installed=true, ml.max_open_jobs=20}}]}" , 
	"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][10.233.92.41:9300][internal:cluster/coordination/join]",
	"Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-0}{OHs2L7FRTFS_o_wPoWEgwA}{l8PW0iCmS82lr96DK6baOg}{10.233.92.41}{10.233.92.41:9300}{ml.machine_memory=4294967296, xpack.installed=true, ml.max_open_jobs=20}",
	"at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) [x-pack-security-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:693) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.1.1.jar:7.1.1]",
	"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
	"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
	"at java.lang.Thread.run(Thread.java:835) [?:?]"] }
	{"type": "server", "timestamp": "2019-06-18T16:58:17,411+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "no known master node, scheduling a retry"  }
	{"type": "server", "timestamp": "2019-06-18T16:58:18,412+0000", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "timed out while retrying [cluster:monitor/health] after failure (timeout [1s])"  }
	{"type": "server", "timestamp": "2019-06-18T16:58:18,413+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}" , 
	"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
	"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.1.1.jar:7.1.1]",
	"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.1.jar:7.1.1]",
	"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
	"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
	"at java.lang.Thread.run(Thread.java:835) [?:?]"] }

Any additional context:

Источник


  • Type:


    Bug

  • Status:

    Resolved


  • Priority:


    Major

  • Resolution:

    Not A Problem


  • Affects Version/s:



    5.1

  • Fix Version/s:



    None

I use solr5.1.0 and deploy one node with solrcloud，and create a collection with 1 shard and 2 replica，when i use solrj to insert data，it throw ”Not enough nodes to handle the request“，but if i create collection with 1 shard and 1 replica，it can insert successfully，also i create another replica with admin api，it still work fine，no longer throw that exception
the full exception stack
Exception in thread «main» org.apache.solr.client.solrj.SolrServerException: org.apache.solr.common.SolrException: Not enough nodes to handle the request
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:929)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:922)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:922)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:922)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:922)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:922)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:782)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:107)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:72)
Caused by: org.apache.solr.common.SolrException: Not enough nodes to handle the request
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1052)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:839)
… 10 more

Источник

Не могу подружить laravel и elasticsearch используя Docker и `babenkoivan/scout-elasticsearch-driver` .
Когда запускаю Docker, все контейнеры, включая elasticsearch, запускаются отлично, но при попытке ввода:

php artisan elastic:create-index "AppTutorialIndexConfigurator"

получаю ошибку:
No alive nodes found in your cluster
Так же, при попытке запроса на 9200 порт проекта через curl в docker workspace контейнере получаю

curl: (7) Failed to connect to localhost port 9200: Connection refused

при этом, если сделать такой же запрос с терминала, то получаю ответ от elasticsearch с json данными, то есть все ок. Я думаю основная проблема как-то с этим может быть связана.

Попробовал запустить такие же контейнеры с laradock — результат такой же.

Уже 3 дня борюсь и сдаюсь, потому очень прошу помочь.

Вот мой docker-compose.yml

spoiler

version: '3.1'
    
    #volumes:
    #  elasticsearch:
    #    driver: local
    
    volumes:
      esdata1:
        driver: local
      esdata2:
        driver: local
      esdata3:
        driver: local
    
    networks:
      esnet:
      frontend:
      backend:
    
    services:
      nginx:
        image: nginx
        ports:
            - "80:80"
            - "443:443"
        volumes:
          - ./hosts:/etc/nginx/conf.d
          - ./www:/var/www
          - ./logs:/var/log/nginx
        links:
          - php
        networks:
          esnet:
          frontend:
            aliases:
              - api.dev
          backend:
            aliases:
              - api.dev
    
      mysql:
        image: mysql:5.7
    
        ports: 
        - "3306:3306"
    
        volumes:
          - ./mysql:/var/lib/mysql
    
        environment:
            MYSQL_ROOT_PASSWORD: secret
    
        networks:
          - esnet
          - frontend
          - backend
    
      # postgres:
      #   image: postgres
    
      #   ports: 
      #     - "3306:3306"
    
      #   environment:
      #       MYSQL_ROOT_PASSWORD: secret
    
      adminer:
        image: adminer
        restart: always
        ports:
          - 8080:8080
    
      php:
        build: ./images/php
    
        links:
          - mysql
    
        volumes:
          - ./www:/var/www
    
        networks:
          - esnet
          - frontend
          - backend
    
      workspace:
          build: ./images/workspace
          volumes:
            - ./www:/var/www:cached
          extra_hosts:
            - "dockerhost:10.0.75.1"
          ports:
            - "2222:22"
          tty: true
          networks:
            - esnet
            - frontend
            - backend
    
      redis:
        image: redis:latest
    
        volumes:
              - ./www/redis:/data
    
        ports:
          - "6379:6379"
    
    
      elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:6.4.2
        container_name: elasticsearch
        environment:
          - cluster.name=docker-cluster
          - bootstrap.memory_lock=true
          - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
          - http.cors.enabled=true
          - http.cors.allow-origin=*
          - discovery.zen.minimum_master_nodes=2
        ulimits:
          memlock:
            soft: -1
            hard: -1
        volumes:
          - esdata1:/usr/share/elasticsearch/data
        ports:
          - 9200:9200
        networks:
          - esnet
          - frontend
          - backend
    
      lasticsearch2:
        image: docker.elastic.co/elasticsearch/elasticsearch:6.4.2
        container_name: elasticsearch2
        environment:
          - cluster.name=docker-cluster
          - bootstrap.memory_lock=true
          - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
          - "discovery.zen.ping.unicast.hosts=elasticsearch"
          - http.cors.enabled=true
          - http.cors.allow-origin=*
          - discovery.zen.minimum_master_nodes=2
        ulimits:
          memlock:
            soft: -1
            hard: -1
        volumes:
          - esdata2:/usr/share/elasticsearch/data
        networks:
          - esnet
          - frontend
          - backend
    
      elasticsearch3:
        image: docker.elastic.co/elasticsearch/elasticsearch:6.4.2
        container_name: elasticsearch3
        environment:
          - cluster.name=docker-cluster
          - bootstrap.memory_lock=true
          - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
          - "discovery.zen.ping.unicast.hosts=elasticsearch"
          - http.cors.enabled=true
          - http.cors.allow-origin=*
          - discovery.zen.minimum_master_nodes=2
        ulimits:
          memlock:
            soft: -1
            hard: -1
        volumes:
          - esdata3:/usr/share/elasticsearch/data
        networks:
          - esnet
          - frontend
          - backend
    
      kibana:
        image: 'docker.elastic.co/kibana/kibana:6.4.2'
        container_name: kibana
        environment:
          SERVER_NAME: kibana.local
          ELASTICSEARCH_URL: http://elasticsearch:9200
        ports:
          - '5601:5601'
        networks:
          - esnet
          - frontend
          - backend
    
      headPlugin:
        image: 'mobz/elasticsearch-head:5'
        container_name: head
        ports:
          - '9100:9100'
        networks:
          - esnet
          - frontend
          - backend

Так же конфиг scout_elastic

return [
        'client' => [
            'hosts' => [
                env('SCOUT_ELASTIC_HOST', 'localhost:9200'),
            ],
        ],
        'update_mapping' => env('SCOUT_ELASTIC_UPDATE_MAPPING', true),
        'indexer' => env('SCOUT_ELASTIC_INDEXER', 'single'),
        'document_refresh' => env('SCOUT_ELASTIC_DOCUMENT_REFRESH'),
    ];

И .env scout конфиг SCOUT_DRIVER=elastic

Источник

EDIT: SOLVED. The latest Ubuntu apparently still ships with Node 10. Nodesource’s Github has instructions on how to add their PPA to install the latest version. Thanks u/phaedrus322 for pointing me in the right direction.

First time trying Laravel. Used to do php development. Never used a framework before. Not sure yet if this is more work or less than doing it by hand.

I’m trying to get Laravel running on the mattrayner/lamp Docker image (using latest) because eventually the goal is to deploy this on a hosting provider and I’d like to see how it performs on a regular LAMP server not optimized by Laravel devs as the Laravel Docker containers surely are. The default index page comes up and I’ve run my first migration.

As suggested in the docs I am trying to install Breeze for authentication features. npm was not installed in the container, but the container seems to be based on Ubuntu so I ran apt update and apt install npm. npm install finishes successfully with warnings:

npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN notsup Unsupported engine for laravel-mix@6.0.39: wanted: {"node":">=12.14.0"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: laravel-mix@6.0.39
npm WARN notsup Unsupported engine for tailwindcss@2.2.19: wanted: {"node":">=12.13.0"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: tailwindcss@2.2.19
npm WARN notsup Unsupported engine for fs-extra@10.0.0: wanted: {"node":">=12"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: fs-extra@10.0.0
npm WARN notsup Unsupported engine for img-loader@4.0.0: wanted: {"node":">=12"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: img-loader@4.0.0
npm WARN notsup Unsupported engine for postcss-loader@6.2.0: wanted: {"node":">= 12.13.0"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: postcss-loader@6.2.0
npm WARN notsup Unsupported engine for webpack-dev-server@4.4.0: wanted: {"node":">= 12.13.0"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: webpack-dev-server@4.4.0
npm WARN notsup Unsupported engine for webpackbar@5.0.2: wanted: {"node":">=12"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: webpackbar@5.0.2
npm WARN notsup Unsupported engine for yargs@17.2.1: wanted: {"node":">=12"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: yargs@17.2.1
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.3.2 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN notsup Unsupported engine for http-proxy-middleware@2.0.1: wanted: {"node":">=12.0.0"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: http-proxy-middleware@2.0.1
npm WARN notsup Unsupported engine for open@8.4.0: wanted: {"node":">=12"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: open@8.4.0
npm WARN notsup Unsupported engine for strip-ansi@7.0.1: wanted: {"node":">=12"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: strip-ansi@7.0.1
npm WARN notsup Unsupported engine for webpack-dev-middleware@5.2.2: wanted: {"node":">= 12.13.0"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: webpack-dev-middleware@5.2.2
npm WARN notsup Unsupported engine for ansi-regex@6.0.1: wanted: {"node":">=12"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: ansi-regex@6.0.1
npm WARN notsup Unsupported engine for schema-utils@4.0.0: wanted: {"node":">= 12.13.0"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: schema-utils@4.0.0
npm WARN notsup Unsupported engine for postcss-nested@5.0.6: wanted: {"node":">=12.0"} (current: {"node":"10.19.0","npm":"6.14.4"})
npm WARN notsup Not compatible with your version of node/npm: postcss-nested@5.0.6

added 810 packages from 459 contributors and audited 813 packages in 33.435s

84 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

But npm run dev dies immediately with this page of errors:

> @ dev /app
> npm run development


> @ development /app
> mix

Error: You are using an unsupported version of Node. Please update to at least Node v12.14
    at assertSupportedNodeVersion (/app/node_modules/laravel-mix/src/Engine.js:6:15)
    at executeScript (/app/node_modules/laravel-mix/bin/cli.js:61:5)
    at Command.program.command.description.option.action (/app/node_modules/laravel-mix/bin/cli.js:47:13)
    at Command.listener [as _actionHandler] (/app/node_modules/commander/index.js:922:31)
    at Command._parseCommand (/app/node_modules/commander/index.js:1503:14)
    at Command._dispatchSubcommand (/app/node_modules/commander/index.js:1443:18)
    at Command._parseCommand (/app/node_modules/commander/index.js:1469:12)
    at Command.parse (/app/node_modules/commander/index.js:1292:10)
    at Command.parseAsync (/app/node_modules/commander/index.js:1318:10)
    at run (/app/node_modules/laravel-mix/bin/cli.js:50:19)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @ development: `mix`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @ development script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-11-22T22_28_43_650Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @ dev: `npm run development`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @ dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-11-22T22_28_43_834Z-debug.log

I can clearly see it thinks I don’t have a new enough version of node.js, but running apt install nodejs tells me:

nodejs is already the newest version (10.19.0~dfsg-3ubuntu1).

Building npm in this Docker needed something like 700 new packages so this must be some amazing auth package to need all that just for some login and registration pages. But I don’t know how to make it work if it won’t install using the newest version of Node available for Ubuntu.

Источник

0/1 nodes available: insufficient cpu, insufficient memory

Overview

Example errors:

0/1 nodes available: insufficient memory
0/1 nodes available: insufficient cpu

More generally:

0/[n] nodes available: insufficient [resource]

This issue happens when Kubernetes does not have enough resources to fulfil your workload request.

Initial Steps Overview

  1. Determine requested resources

  2. Have you requested too many resources?

Detailed Steps

1) Determine requested resources

To determine your requested resources for your workload, you must first extract its YAML.

What type of resource to extract the YAML for may depend, but most commonly you can just get the YAML for the pod that reports the problem.

From that YAML, determine whether there are any resource requests made in the containers section, under resources.

A simplified YAML that makes a large request for memory resources (1000G) might look like this, for example:

apiVersion: v1
kind: Pod
metadata:
  name: too-much-mem
spec:
  containers:
    - command:
        - sleep
        - "3600"
      image: busybox
      name: broken-pods-too-much-mem-container
      resources:
        requests:
          memory: "1000Gi"

If no resource requests are in the YAML, then a default request may be made. What this request is will depend on other configuration in the cluster. See here for more information.

See here for more information.

If no request is made and you are out of resources, then it is likely that you have no available nodes. At this point you need to consider solution A.

2) Have you requested too many resources?

If you have made a resource request, then there are two possibilities:

  • Your resource request cannot fit into any node on the cluster

  • Your resource request can fit on a node in the cluster, but those nodes already have workloads running on them which block yours being provisioned

Step 1 should have shown you whether you are specifically requesting resources. Once you know what those resources are, you can compare them to the resources available on each node.

If you are able, run:

which, under ‘Capacity:’, ‘Allocatable:’, and ‘Allocated resources:’ will tell you the resources available in each node, eg:

$ kubectl describe nodes
[...]
Capacity:
  cpu:                4
  ephemeral-storage:  61255492Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             2038904Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  56453061334
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             1936504Ki
  pods:               110
[...]
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                750m (18%)  0 (0%)
  memory             140Mi (7%)  340Mi (17%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)

You should be able to compare these to the resources you requested to determine why your request was not met, and choose where to autoscale or provision a larger node accordingly.

Link to Solution A

Solutions List

A) Set up autoscaling

B) Provision appropriately-sized nodes

Solutions Detail

A) Set up autoscaling

The details of this will vary depending on your platform, but in general the principle is that you have legitimately used up all your resources, and you need more nodes to take the load.

Note this solution will not work if:

  • Your nodes are unavailable for other reasons (such as: you have a ‘runaway’ workload that is consuming all the resources it finds), as you will see this error again once the new resources are consumed.

  • Your workload cannot fit on any node in the cluster

Some potentially useful links to achieving this:

  • K8s autoscaling
  • EKS
  • GKE
  • AKS

B) Provision appropriately-sized nodes

The details of this will vary according to your platform. You will need to add a node (or set of nodes) that exceeds in size the amount your workload is requesting.

Also note: your workload scheduler may ‘actively’ or ‘intelligently’ move workloads to make them all ‘fit’ onto the given nodes. In these cases, you may need to significantly over-provision node sizes to reliably accommodate your workload.

Check Resolution

If the error is no longer seen in the workload description in Kubernetes, then this particular issue has been resolved.

Further Information

When you make a request for Kubernetes to run your workload, it tries to find all the nodes that can fulfil the requirements.

Kubernetes resource management docs

Owner

email

comments powered by
Источник

Понравилась статья? Поделить с друзьями:
  • Validation error messages
  • Validation error for column перевод
  • Validation error for column id value null
  • Validation error flask
  • Validation error drf