Error keepererrorcode connectionloss for hbase master

I’m going completely crazy:

Installed Hadoop/Hbase, all is running;

/opt/jdk1.6.0_24/bin/jps
23261 ThriftServer
22582 QuorumPeerMain
21969 NameNode
23500 Jps
23021 HRegionServer
22211 TaskTracker
22891 HMaster
22117 SecondaryNameNode
21779 DataNode
22370 Main
22704 JobTracker

Pseudo distributed environment.

hbase shell

is working and coming up with correct results running ‘list’ and;

hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011

hbase(main):001:0> status
1 servers, 0 dead, 8.0000 average load

When connecting via ruby & thrift, everything is working fine; we are adding data, it’s getting in the system, we can query/scan it. Everything seems fine.

However, when connecting with Java:

groovy> import org.apache.hadoop.hbase.HBaseConfiguration 
groovy> import org.apache.hadoop.hbase.client.HBaseAdmin 
groovy> conf = HBaseConfiguration.create() 
groovy> conf.set("hbase.master","127.0.0.1:60000"); 
groovy> hbase = new HBaseAdmin(conf); 

Exception thrown

org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1000)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:294)
    at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:84)

I’ve been trying to find the cause, but I really have no clue at all. Everything seems to be correctly installed.

netstat -lnp|grep 60000
tcp6       0      0 :::60000                :::*                    LISTEN      22891/java  

Looks fine as well.

# telnet localhost 60000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connects and dies if you type anything + enter (not sure if that’s the idea, thrift on 9090 does the same).

Can anyone help me?

errorlog.txtThis thread is related to my post related to Metrics collector not starting (https://community.hortonworks.com/questions/23512/metrics-collector-is-not-starting-showing-error-re.html). I thought of creating a new thread to highlight the main issue that I see in the log files. In general, Metric Collector is not starting. The logs have many information related to the error. The error that is mainly shown

exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master

In the same log file, I also see the following line

zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:61181.

If ‘localhost’ is the issue here, then I am not sure why localhost is used here instead of FQDN and I do not know which configuration file will help me to use FQN instead of localhost.

I have read many threads related to this error but none of them helped me fix this issue. Can anyone please help me understand this issue.

I am pasting the value some of the properties:

hbase.rootdir = hdfs://item-70288:8020/apps/hbase/data

hbase.cluster.distributed= true

Metrics service operation mode = embedded

hbase.zookeeper.property.clientPort= 2181

hbase.zookeeper.quorum =item-70288

Update #1

=========

The command netstat -anp |grep 61181 is not returning anything. It seems nothing is listening on 61181.

I am attaching the full error log with this post.

Support Questions

• Subscribe to RSS Feed
• Mark Question as New
• Mark Question as Read
• Float this Question for Current User
• Bookmark
• Subscribe
• Mute
• Printer Friendly Page

Created ‎03-18-2016 10:14 AM

• Mark as New
• Bookmark
• Subscribe
• Mute
• Subscribe to RSS Feed
• Permalink
• Print
• Report Inappropriate Content

errorlog.txtThis thread is related to my post related to Metrics collector not starting (https://community.hortonworks.com/questions/23512/metrics-collector-is-not-starting-showing-error-re.html). I thought of creating a new thread to highlight the main issue that I see in the log files. In general, Metric Collector is not starting. The logs have many information related to the error. The error that is mainly shown

exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master

In the same log file, I also see the following line

zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:61181 .

If ‘localhost’ is the issue here, then I am not sure why localhost is used here instead of FQDN and I do not know which configuration file will help me to use FQN instead of localhost.

I have read many threads related to this error but none of them helped me fix this issue. Can anyone please help me understand this issue.

I am pasting the value some of the properties:

Metrics service operation mode = embedded

The command netstat -anp |grep 61181 is not returning anything. It seems nothing is listening on 61181.

I am attaching the full error log with this post.

Created ‎03-22-2016 08:15 AM

• Mark as New
• Bookmark
• Subscribe
• Mute
• Subscribe to RSS Feed
• Permalink
• Print
• Report Inappropriate Content

I have fixed the issue related to Metrics Monitor not starting. Even though I have no idea why the above error came, I fixed by changing/setting the following property under Ambari Metrics Monitor configuration through Ambari.

1. set hbase.zookeeper.property.clientPort to 2181. I had observed that this was set to 61181, which I am not sure how it happend. This is a very important property. The Zookeeper server also runs at the same port, so I guess the hbase.zookeeper.property.clientPort should also be set to the same port. If these ports are different, then there will be a number of errors thrown by Ambari-metrics collector service is started. Therefore make sure this property is set to the same port as the zookeeper server.
2. Change the «Metrics Service operation mode» to distributed.
3. Set hbase.cluster.distributed to true
4. Set hbase.rootdir to a hdfs folder. I created a new folder in the hdfs root and made hdfs the owner of the folder.

After I did this, I restarted only the Ambari Metric collector and this time it was happy to start with a green flag 😉

Источник

Support Questions

• Subscribe to RSS Feed
• Mark Question as New
• Mark Question as Read
• Float this Question for Current User
• Bookmark
• Subscribe
• Mute
• Printer Friendly Page

Created ‎03-18-2016 10:14 AM

• Mark as New
• Bookmark
• Subscribe
• Mute
• Subscribe to RSS Feed
• Permalink
• Print
• Report Inappropriate Content

errorlog.txtThis thread is related to my post related to Metrics collector not starting (https://community.hortonworks.com/questions/23512/metrics-collector-is-not-starting-showing-error-re.html). I thought of creating a new thread to highlight the main issue that I see in the log files. In general, Metric Collector is not starting. The logs have many information related to the error. The error that is mainly shown

exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master

In the same log file, I also see the following line

zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:61181 .

If ‘localhost’ is the issue here, then I am not sure why localhost is used here instead of FQDN and I do not know which configuration file will help me to use FQN instead of localhost.

I have read many threads related to this error but none of them helped me fix this issue. Can anyone please help me understand this issue.

I am pasting the value some of the properties:

Metrics service operation mode = embedded

The command netstat -anp |grep 61181 is not returning anything. It seems nothing is listening on 61181.

I am attaching the full error log with this post.

Created ‎03-22-2016 08:15 AM

• Mark as New
• Bookmark
• Subscribe
• Mute
• Subscribe to RSS Feed
• Permalink
• Print
• Report Inappropriate Content

I have fixed the issue related to Metrics Monitor not starting. Even though I have no idea why the above error came, I fixed by changing/setting the following property under Ambari Metrics Monitor configuration through Ambari.

1. set hbase.zookeeper.property.clientPort to 2181. I had observed that this was set to 61181, which I am not sure how it happend. This is a very important property. The Zookeeper server also runs at the same port, so I guess the hbase.zookeeper.property.clientPort should also be set to the same port. If these ports are different, then there will be a number of errors thrown by Ambari-metrics collector service is started. Therefore make sure this property is set to the same port as the zookeeper server.
2. Change the «Metrics Service operation mode» to distributed.
3. Set hbase.cluster.distributed to true
4. Set hbase.rootdir to a hdfs folder. I created a new folder in the hdfs root and made hdfs the owner of the folder.

After I did this, I restarted only the Ambari Metric collector and this time it was happy to start with a green flag 😉

Источник

Zookeeper Connection Loss Errors

You’re viewing Apigee Edge documentation.
View Apigee X documentation.

Symptom

The ZooKeeper connectivity issues can manifest as different symptoms such as:

1. API proxy deployment errors
2. Management API calls fail with 5XX errors
3. Routers or Message Processors fail to start
4. Analytics components report ZooKeeper connection loss in system.logs

Error messages

The following provides examples of error messages that may be observed when there is connection loss to ZooKeeper node(s).

1. The following error is returned in Management Server logs when an API Proxy deployment fails due to ZooKeeper Connection loss:
2. During startup, the Routers and Message Processors connect to ZooKeeper. If there are connectivity issues with ZooKeeper, then these components will fail to start with the following error:
3. The Edge UI may display the following error indicating it was unable to check the deployment status of the API Proxies:

Possible causes

The following table lists possible causes of this issue:

Click a link in the table to see possible resolutions to that cause.

Network connectivity issue across different data centers

Diagnosis

A ZooKeeper cluster may have nodes that span across multiple regions/data centers, such as DC-1 and DC-2. The typical Apigee Edge 2 DC topology will have:

• ZooKeeper servers 1, 2, and 3 as voters in DC-1
• ZooKeeper 4 and 5 as voters and ZooKeeper 6 as an observer in DC-2.

If DC-1 region goes down or network connectivity between DC-1 and DC-2 is broken, then ZooKeeper nodes cannot elect a new leader in DC-2 and they fail to communicate with the leader node. ZooKeeper observers cannot elect a new leader and the two remaining voters in DC-2 do not have a quorum of at least 3 voter nodes to elect a new leader. Thus, the ZooKeepers in DC-2 will not be able to process any requests. The remaining ZooKeeper nodes in DC-2 will continue to loop retrying to connect back to the ZooKeeper voters to find the leader.

Resolution

Apply the following solutions to address this issue in the specified order.

If you are unable to resolve the problem after trying these solutions, please contact Apigee Support.

Solution #1

1. Work with your network administrators to repair the network connectivity issue between the data centers.
2. When the ZooKeeper ensemble are able to communicate across the data centers and elect a ZooKeeper leader, the nodes should become healthy and be able to process requests.

Solution #2

1. If the network connectivity will take time to repair, a workaround is to reconfigure ZooKeeper nodes in the region where they are down. For example, reconfigure the ZooKeeper cluster in DC-2 so that the 3 ZooKeeper nodes in this region are all voters and remove the server.# in the zoo.cfg of the ZooKeepers from DC-1 region.
   1. In the following example, zoo.cfg configures nodes for 2 regions where DC-1 uses us-ea hostnames denoting US-East region and DC-2 uses us-wo hostnames denoting US-West region. (NOTE: Only relevant configs are displayed):

   In the above example, reconfigure the zoo.cfg as follows:

2. Using code with config, create a file /opt/apigee/customer/application/zookeeper.properties with the following:

In the above, the nodes from US-East are removed, and the US-West nodes get promoted to voters when the :observer annotation is removed.

Back up /opt/apigee/apigee-zookeeper/conf/zoo.cfg and old /opt/apigee/customer/application/zookeeper.properties .

These files will be used to restore the defaults when the network connectivity is back up between data centers.

Disable the observer notation for the observer node. To do this, add the following configuration to the top of /opt/apigee/customer/application/zookeeper.properties :

Edit the /opt/apigee/data/apigee-zookeeper/data/myid file as follows:

• For server.1 , change the entry inside myid from 4 to 1.
• For server.2 , change the myid from 5 to 2.
• For server.3 change the myid from 6 to 3.

ZooKeeper node not serving requests

A ZooKeeper node in the ensemble may become unhealthy and be unable to respond to client requests. This could be because:

1. The node was stopped without being restarted.
2. The node was rebooted without auto-start enabled.
3. System load on the node caused it to go down or become unhealthy.

Diagnosis

1. Execute the following ZooKeeper health check commands on each of the ZooKeeper nodes and check the output:
   1. Note:NOTE: Response “imok” is a successful response.

   Check the mode to determine if the ZooKeeper node is a leader or follower.

   Example output for an all in one, single ZooKeeper node:

   This command lists the ZooKeeper variables which can be used to check the health of the ZooKeeper cluster.

   This command lists statistics about performance and connected clients.

   This command gives extended details on ZooKeeper connections.

   If any of the last 3 health check commands show the following message:

   Then it indicates that specific ZooKeeper node(s) is not serving requests.

Resolution

1. Restart all other ZooKeeper nodes in the cluster one by one.
2. Re-run the ZooKeeper health check commands on each node and see if you get the expected output.

Contact Apigee Support to troubleshoot the cause of system load if it persists or if restarts does not resolve the problem.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Источник

Ошибка Hbase «ОШИБКА: KeeperErrorCode = NoNode для / hbase / master»

При выполнении любой команды в оболочке hbase я получаю следующую ошибку «ERROR: KeeperErrorCode = NoNode for / hbase / master» в оболочке hbase.

Запущен HBASE:

При проверке статуса в HBASE SHELL:

hbase-site.xml

Пожалуйста, дайте мне знать, почему эта ошибка возникает при выполнении команд hbase?

Во-первых, убедитесь, что сопоставление IP-адреса и имени хоста настроено в файле hosts.

Во-вторых, измените расположение временного каталога HBase. данные временного каталога будут регулярно очищаться. По умолчанию временный каталог находится на /tmp , измените их в hbase-site.xml .

Если не работает. очистите каталог данных hbase, также очистите метаданные в zookeeper, перезапустите hbase снова.

Более того, проверьте свой NTP и брандмауэр.

Спасибо за ответ. Я создал каталог / hbase / tmp и настроил это свойство в hbase-site.xml, очистил оба данных, каталог zookeeper. Но по-прежнему получаю то же исключение.

@ user7413163, если вам нужна дополнительная помощь, покажите свои журналы, не только ошибки

замените или добавьте эту конфигурацию в файл hbase-site.xml в папке conf каталога hbase, а затем повторно запустите команду «hbase shell», а затем команду «list», чтобы просмотреть имеющиеся таблицы.

Я знаю, что это не связано с искрой, но я получал следующие ошибки;

Вызвано: java.io.IOException: org.apache.zookeeper.KeeperException $ NoNodeException: KeeperErrorCode = NoNode для / hbase / meta-region-server

Вызвано: org.apache.zookeeper.KeeperException $ NoNodeException: KeeperErrorCode = NoNode для / hbase / meta-region-server

Вызвано: org.apache.zookeeper.KeeperException $ NoNodeException: KeeperErrorCode = NoNode для / hbase / hbasei

Установка конфигурации hbase.rootdir решила мои проблемы при создании HBaseContext;

Так что вы можете попробовать добавить эту конфигурацию в hbase-site.xml.

В моем случае я получал этот « ERROR: KeeperErrorCode = NoNode for /hbase/master », потому что процесс HMaster не работал.

Проверьте с помощью команды jps .

Если вы не видите HMaster процесс, как в приведенном выше списке, то это причина ERROR: KeeperErrorCode = NoNode. в оболочке hbase.

Источник