Error uncaught exception in thread kafka admin client thread adminclient 1

KAFKA и SSL: java.lang.OutOfMemoryError: пространство кучи Java при использовании команды kafka-themes в кластере SSL KAFKA

Это мой первый пост на Stackoverflow, надеюсь, я выбрал не тот раздел.

Размер Kafka HEAP настраивается для следующего файла:

Со следующим параметром:

ОС «CentOS Linux выпуск 7.7.1908» .

Kafka — это «confluent-kafka-2.12-5.3.1-1.noarch» , установленный из следующего репозитория:

Я активировал SSL на 3-машинном кластере KAFKA несколько дней назад, и вдруг следующая команда перестала работать:

Которые возвращают мне следующую ошибку:

В журнале сервера появляется следующая строка, когда я пытаюсь запросить его через «kafka-themes»:

Я смог правильно использовать эту команду ДО внедрения SSL в кластере. Вот конфигурация, которую я использую. Все функции работают должным образом (потребители, производители . ), кроме «Кафка-топики»:

Нет проблем с сертификатом (который подписан внутренним CA, внутренним CA, который я добавил в склад доверенных сертификатов, указанный в конфигурации). OpenSSL не показывает ошибок:

Следующая команда очень хорошо работает с SSL, благодаря параметру «-consumer.config client-ssl.properties»

Сейчас я вынужден использовать «—zookeeper», который, согласно документации, устарел:

И, конечно, это работает довольно хорошо:

Итак, мой вопрос: почему я не могу использовать «—bootstrap-server» банкомат? Из-за устаревшего «zookeeper» я беспокоюсь, что не смогу проконсультироваться с моими темами и их деталями .

Я считаю, что для kafka-themes нужна та же опция, что и для kafka-console-consumer, или «-consumer.config» .

Спросите, нужна ли дополнительная точность.

Большое спасибо, надеюсь, мой вопрос ясен и читаем.

Источник

KAFKA and SSL : java.lang.OutOfMemoryError: Java heap space when using kafka-topics command on KAFKA SSL cluster

this is my first post on Stackoverflow, i hope i didnt choose the wrong section.

Kafka HEAP size is configured on following file :

With following parameter :

OS is «CentOS Linux release 7.7.1908».

Kafka is «confluent-kafka-2.12-5.3.1-1.noarch», installed from the following repository :

I activated SSL on a 3-machine KAFKA cluster few days ago, and suddently, the following command stopped working :

Which return me the following error :

On the server’s log, the following line appears when i try to request it via «kafka-topics» :

I was able to use this command properly BEFORE implementing SSL on the cluster. Here is the configuration i’m using. All functionnality work properly (consumers, producers. ) except «kafka-topics» :

There is no problem with the certificate (which is signed by internal CA, internal CA which i added to the truststore specified on the configuration). OpenSSL show no errors :

The following command is working pretty well with SSL, thanks to parameter «-consumer.config client-ssl.properties»

«client-ssl.properties» content is :

Right now, i’m forced to use «—zookeeper», which according to the documentation, is deprecated :

And of course, it’s working pretty well :

So, my question is : why am i unable to use «—bootstrap-server» atm ? Because of the «zookeeper» deprecation, i’m worried about not to be able to consult my topics, and their details.

I believe that kafka-topics needs the same option than kafka-console-consumer, aka «-consumer.config».

Ask if any additionnal precision needed.

Thanks a lot, hope my question is clear and readable.

Источник

marwei / how_to_reset_kafka_consumer_group_offset.md

Kafka 0.11.0.0 (Confluent 3.3.0) added support to manipulate offsets for a consumer group via cli kafka-consumer-groups command.

1. List the topics to which the group is subscribed

Note the values under «CURRENT-OFFSET» and «LOG-END-OFFSET». «CURRENT-OFFSET» is the offset where this consumer group is currently at in each of the partitions.

1. Reset the consumer offset for a topic (preview)

This will print the expected result of the reset, but not actually run it.

1. Reset the consumer offset for a topic (execute)

This will execute the reset and reset the consumer group offset for the specified topic back to 0.

1. Repeat 1 to check if the reset is successful

• The consumer group must have no running instance when performing the reset. Otherwise the reset will be rejected.
• There are many other resetting options, run kafka-consumer-groups for details
  • —shift-by
• —to-current
• —to-latest
• —to-offset
• —to-datetime
• —by-duration

For ones who want to do the same for Kafka 0.10 you may download Kafka 1.0 binaries which include the script with this new offset reset capabilities, it will work for Kafka 0.10 as well:

thx for instruction 🙂 Do you know why it is so hard to achieve in most libraries? Did you do any research about it? I have checked that in kafka-go, sarama ( both golang) and spring-kafka — there is no easy way to reset offset while using consumer groups. I just have problem with understanding if its dangerous to reset offset or there are some other technical issues.

thanks daniilyar. but we’re facing below error:

Error: Executing consumer group command failed due to The broker only supports OffsetFetchRequest v1, but we need v2 or newer to request all topic partitions.

Seems like version incompatibility here to execute commands with 1.0 binaries.

Anyone know if there’s a way to do this on Kafka 0.9.0 ?

Apparently my latests and my earliests offsets have the same values, and are equal to LOG-END-OFFSET. I can not manage to go back to the beginning of the messages.

kafka .10 does not support —rest-offsets but this can be achieved using java apis.
Please have a look.

Can anyone tell me how to do this for kafka 1.0.0

The API is all the same since 0.11. This includes 1.0.0, 2.0, and above until anyone mentions otherwise.

Applying 1.0 to 0.10 didn’t work for me either. Same as @Joginder22:
Error: Executing consumer group command failed due to The broker only supports OffsetFetchRequest v1, but we need v2 or newer to request all topic partitions.

I would like to make a small adjustment about how to reset a topic partition

kafka1.1 doesn’t seem to support reset by —to-datetime. it can effect —to-earliest but do not effect —to-datetime

Anyone knows if it’s possible to reset the group offset using librdkafkacpp APIs? Thanks

Just leaving a note here to save everyone using a Kafka cluster protected by SSL time. In case you are getting the following OOM error when running kafka-consumer-groups :

Counterintuitively, you are not running out of memory, but rather this arises as a result of not providing SSL credentials. You need to create a .properties file and pass it into each call to the binary via the —command-config option. In our case, the template for this properties file is:

Hope this saves someone time in the future.

Источник

ERROR Uncaught exception in herder work thread in Distributed mode #189

I am trying to start Kafka connect (Confluent 3.1.1) in distributed mode but I am having some problems:

[2017-02-01 11:08:52,803] ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:181)
java.lang.NullPointerException
at org.apache.kafka.connect.storage.KafkaConfigBackingStore$ConsumeCallback.onCompletion(KafkaConfigBackingStore.java:444)
[2017-02-01 11:08:51,893] INFO Discovered coordinator inaki-P552LA:9092 (id: 2147483647 rack: null) for group test. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:555)
[2017-02-01 11:08:52,803] ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:181)
java.lang.NullPointerException
at org.apache.kafka.connect.storage.KafkaConfigBackingStore$ConsumeCallback.onCompletion(KafkaConfigBackingStore.java:444)
at org.apache.kafka.connect.storage.KafkaConfigBackingStore$ConsumeCallback.onCompletion(KafkaConfigBackingStore.java:424)
at org.apache.kafka.connect.util.KafkaBasedLog.poll(KafkaBasedLog.java:253)
at org.apache.kafka.connect.util.KafkaBasedLog.readToLogEnd(KafkaBasedLog.java:293)
at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:143)
at org.apache.kafka.connect.storage.KafkaConfigBackingStore.start(KafkaConfigBackingStore.java:259)
at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:114)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:169)
at java.lang.Thread.run(Thread.java:745)
[2017-02-01 11:08:52,804] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:68)
[2017-02-01 11:08:52,804] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:154)
[2017-02-01 11:08:52,807] INFO Stopped ServerConnector@e3cee7b <0.0.0.0:8083>(org.eclipse.jetty.server.ServerConnector:306)
[2017-02-01 11:08:52,813] INFO Stopped o.e.j.s.ServletContextHandler@120f38e6 (org.eclipse.jetty.server.handler.ContextHandler:865)

I did some steps mentioned and If I start workers before connect it works without NullPointerException, but when I send messages throw timeout errors

The text was updated successfully, but these errors were encountered:

What version are you on and did you resolve this? We did used to see this sometimes and I think we tried to add some logging to give more info if it happened again because it was difficult to tell how things had gotten out of sync from looking at the code.

I could’t resolve it, I just restart everything to default following documentation and test again. So, the error disappeared

Reproducing this consistently with Kafka Connect 1.1.0 against a 1.0.0 cluster. Repro steps are:

1. Shutdown Kafka connect
2. Delete Kafka Connect topics (offsets, status, configs)
3. (assuming auto-creation enabled in cluster) Start Kafka Connect back up
4. Send GET request to https://my-kafka-connect/connectors

Symptoms, you’ll get a timeout on the GET request and an exception like this:

Interestingly, it will succeed in creating the offsets topic, but not the config or status topics.

Resetting cluster repeatedly (by deleting Kafka Connect topics) does not resolve.

This happened immediately after upgrading Kafka Connect from 1.0 to 1.1 . No other changes to cluster or deployment environment. @ewencp

I may have just figured this out. it looks like the status topic has been eternally marked for deletion (confirming from within ZK). It’s no longer listed in the broker metadata as an existing topic but it appears ZK never completed the delete. I think this causing a strange state where when Kafka Connect then tries to recreate it, Kafka can’t service the request because ZK disagrees. not sure if this is the same root cause as the original reported issue but it’s something.

We ended up forcing the controllers to reset their «deleting state» of the Kafka Connect topics by:

1. Shutdown Kafka connect
2. Running deletes from kafka-topics.sh
3. Deleting ZK entries
4. Creating topics (will appear to fail if controllers are still in «deleting» state but that’s ok)
5. Run deletes again from kafka-topics.sh (ensures resetting controllers «deleting» state)
6. Restart Kafka Connect to automatically create topics for real

Had the same bug, kafka-connect was creating the 3 partitions connect-configs , connect-offsets and connect-status with the default number of partitions (in my case 64 as defined in the kafka server properties) and was causing this issue.

As the above solution, I ended up removing the topics, recreating them with 1 partition and restarting kafka-connect.

I am unable to start kafka connect
See in the kafka connect logs :

Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:253)

org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

[2019-12-25 11:36:31,409] INFO ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = earliest
bootstrap.servers = [localhost:9092]
check.crcs = true
client.dns.lookup = default
client.id =
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = connect-cluster
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
(org.apache.kafka.clients.consumer.ConsumerConfig:347)
[2019-12-25 11:36:31,448] WARN The configuration ‘config.storage.topic’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,448] WARN The configuration ‘status.storage.topic’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,448] WARN The configuration ‘plugin.path’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,449] WARN The configuration ‘internal.key.converter.schemas.enable’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,449] WARN The configuration ‘config.storage.replication.factor’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,449] WARN The configuration ‘value.converter.schema.registry.url’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,449] WARN The configuration ‘internal.key.converter’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,449] WARN The configuration ‘status.storage.replication.factor’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,449] WARN The configuration ‘internal.value.converter.schemas.enable’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,449] WARN The configuration ‘internal.value.converter’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,449] WARN The configuration ‘offset.storage.replication.factor’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,450] WARN The configuration ‘offset.storage.topic’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,450] WARN The configuration ‘value.converter’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,450] WARN The configuration ‘key.converter’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,450] WARN The configuration ‘key.converter.schema.registry.url’ was supplied but isn’t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:355)
[2019-12-25 11:36:31,450] INFO Kafka version: 5.3.0-ccs (org.apache.kafka.common.utils.AppInfoParser:117)
[2019-12-25 11:36:31,450] INFO Kafka commitId: 6481debc2be778ee (org.apache.kafka.common.utils.AppInfoParser:118)
[2019-12-25 11:36:31,451] INFO Kafka startTimeMs: 1577273791450 (org.apache.kafka.common.utils.AppInfoParser:119)
[2019-12-25 11:36:31,533] INFO [Producer clientId=producer-1] Cluster ID: 3gSpLKOtT8y-axt0zWl0oA (org.apache.kafka.clients.Metadata:261)
[2019-12-25 11:36:31,634] INFO [Consumer clientId=consumer-1, groupId=connect-cluster] Cluster ID: 3gSpLKOtT8y-axt0zWl0oA (org.apache.kafka.clients.Metadata:261)
Dec 25, 2019 11:36:32 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.RootResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
Dec 25, 2019 11:36:32 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will be ignored.
Dec 25, 2019 11:36:32 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be ignored.
Dec 25, 2019 11:36:32 AM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

[2019-12-25 11:36:32,532] INFO Started o.e.j.s.ServletContextHandler@43045f9f (org.eclipse.jetty.server.handler.ContextHandler:855)
[2019-12-25 11:36:32,532] INFO REST resources initialized; server is started and ready to handle requests (org.apache.kafka.connect.runtime.rest.RestServer:233)
[2019-12-25 11:36:32,532] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:56)
[2019-12-25 11:37:31,457] ERROR [Worker clientId=connect-1, groupId=connect-cluster] Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:253)
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
[2019-12-25 11:37:31,504] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:66)
[2019-12-25 11:37:31,519] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:241)
[2019-12-25 11:37:31,552] INFO Stopped http_8083@24361cfc <0.0.0.0:8083>(org.eclipse.jetty.server.AbstractConnector:341)
[2019-12-25 11:37:31,571] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:167)
[2019-12-25 11:37:31,706] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:258)
[2019-12-25 11:37:31,706] INFO [Worker clientId=connect-1, groupId=connect-cluster] Herder stopping (org.apache.kafka.connect.runtime.distributed.DistributedHerder:533)
[2019-12-25 11:37:36,749] INFO [Worker clientId=connect-1, groupId=connect-cluster] Herder stopped (org.apache.kafka.connect.runtime.distributed.DistributedHerder:553)
[2019-12-25 11:37:36,749] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:71)

I had the same issue. Resolved by deleting the existing topics connect-configs , connect-status , connect-offsets and recreating with number of partitions =1 for all. (As suggested by @gmanolache )

I had the same issue. Resolved by deleting the existing topics connect-configs , connect-status , connect-offsets and recreating with number of partitions =1 for all. (As suggested by @gmanolache )

Apologies if this is obvious, but for larger or production grade workloads you will need more than 1 partition on some of those topics. This deletion state issue notwithstanding, just thought I should mention that.

I am deployed kafka, zookeeper, kafka-connect using Strimzi in kuberentes. I updated kafka-connect image in kubernetes for RabbitMQSourceConnector plugin and it was working initially, after restart Kafka-Connect we get following error in Kafka-Connect. May I know the reason why it should give below error?

Источник

Issue:- 

When trying to delete the topic in the Amazon MSK kafka clusterr got the following error

Error:- 

java.lang.OutOfMemoryError: Java heap space

 ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1': (org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
	at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
	at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1131)

 kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml  
 

./kafka-topics.sh --bootstrap-server b-1.test-kafka.q15lx0.c10.kafka.us-west-2.amazonaws.com:9094,b-2.test-kafka.q15lx0.c10.kafka.us-west-2.amazonaws.com:9094,b-3.test-kafka.q15lx0.c10.kafka.us-west-2.amazonaws.com:9094 --delete --topic <topic-name>  --command-config  /Users/amittal/kafka/kafka_2.12-2.2.1/bin/client.properties

KAFKA и SSL: java.lang.OutOfMemoryError: пространство кучи Java при использовании команды kafka-themes в кластере SSL KAFKA

Это мой первый пост на Stackoverflow, надеюсь, я выбрал не тот раздел.

Размер Kafka HEAP настраивается для следующего файла:

Со следующим параметром:

ОС «CentOS Linux выпуск 7.7.1908» .

Kafka — это «confluent-kafka-2.12-5.3.1-1.noarch» , установленный из следующего репозитория:

Я активировал SSL на 3-машинном кластере KAFKA несколько дней назад, и вдруг следующая команда перестала работать:

Которые возвращают мне следующую ошибку:

В журнале сервера появляется следующая строка, когда я пытаюсь запросить его через «kafka-themes»:

Я смог правильно использовать эту команду ДО внедрения SSL в кластере. Вот конфигурация, которую я использую. Все функции работают должным образом (потребители, производители . ), кроме «Кафка-топики»:

Нет проблем с сертификатом (который подписан внутренним CA, внутренним CA, который я добавил в склад доверенных сертификатов, указанный в конфигурации). OpenSSL не показывает ошибок:

Следующая команда очень хорошо работает с SSL, благодаря параметру «-consumer.config client-ssl.properties»

Сейчас я вынужден использовать «—zookeeper», который, согласно документации, устарел:

И, конечно, это работает довольно хорошо:

Итак, мой вопрос: почему я не могу использовать «—bootstrap-server» банкомат? Из-за устаревшего «zookeeper» я беспокоюсь, что не смогу проконсультироваться с моими темами и их деталями .

Я считаю, что для kafka-themes нужна та же опция, что и для kafka-console-consumer, или «-consumer.config» .

Спросите, нужна ли дополнительная точность.

Большое спасибо, надеюсь, мой вопрос ясен и читаем.

Источник

kafka-rest java Heap space issue #594

Comments

varun-tangoit commented Nov 4, 2019 •

I’m currently implementing kafka_rest proxy in ec2 instance for aws managed kafka. I got error like
ERROR Uncaught exception in thread ‘kafka-admin-client-thread | adminclient-1’: (org.apache.kafka.common.utils.KafkaThread:51)
java.lang.OutOfMemoryError: Java heap space

Eventhough i improved or set environment variable, can’t able to get result of list topics for that below curl;
curl «http://localhost:8082/topics»
export KAFKA_HEAP_OPTS=»-Xms512m -Xmx1g»

can you please help me, how can i resolve this issue.

The text was updated successfully, but these errors were encountered:

danielsinai commented Dec 21, 2019

try KAFKAREST_HEAP_OPTS instead 🙂

source bin/kafka-rest-run-class:
exec $JAVA **$KAFKAREST_HEAP_OPTS** $KAFKAREST_JVM_PERFORMANCE_OPTS $KAFKAREST_JMX_OPTS $KAFKAREST_LOG4J_OPTS -cp $CLASSPATH $KAFKAREST_OPTS «$MAIN» «$@»

ldicarlo commented Jan 14, 2020

Shouldn’t it be KAFKA_REST + _HEAP_OPTS as stated here ?

ldicarlo commented Jan 14, 2020

Oh, actually on Kafka connect for example, it is taken in account only if I pass KAFKA_HEAP_OPTS , I think

psalot commented May 20, 2020

I had come across the same issue but for me the reason was incomplete configuration for connecting to kafka brokers.
This error occurs even if any of your configuration mismatch has happened.So first validate if your kafka instance has any authentication/authorization enabled.
In my case TLS was enabled so I had to pass client.security.protocol=SSL and client.ssl.truststore.location= property.

Footer

© 2023 GitHub, Inc.

You can’t perform that action at this time.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.

Источник

marwei / how_to_reset_kafka_consumer_group_offset.md

Kafka 0.11.0.0 (Confluent 3.3.0) added support to manipulate offsets for a consumer group via cli kafka-consumer-groups command.

1. List the topics to which the group is subscribed

Note the values under «CURRENT-OFFSET» and «LOG-END-OFFSET». «CURRENT-OFFSET» is the offset where this consumer group is currently at in each of the partitions.

1. Reset the consumer offset for a topic (preview)

This will print the expected result of the reset, but not actually run it.

1. Reset the consumer offset for a topic (execute)

This will execute the reset and reset the consumer group offset for the specified topic back to 0.

1. Repeat 1 to check if the reset is successful

• The consumer group must have no running instance when performing the reset. Otherwise the reset will be rejected.
• There are many other resetting options, run kafka-consumer-groups for details
  • —shift-by
• —to-current
• —to-latest
• —to-offset
• —to-datetime
• —by-duration

daniilyar commented Feb 22, 2018 •

For ones who want to do the same for Kafka 0.10 you may download Kafka 1.0 binaries which include the script with this new offset reset capabilities, it will work for Kafka 0.10 as well:

michalkarolik commented Mar 21, 2018

thx for instruction 🙂 Do you know why it is so hard to achieve in most libraries? Did you do any research about it? I have checked that in kafka-go, sarama ( both golang) and spring-kafka — there is no easy way to reset offset while using consumer groups. I just have problem with understanding if its dangerous to reset offset or there are some other technical issues.

Joginder22 commented Apr 17, 2018

thanks daniilyar. but we’re facing below error:

Error: Executing consumer group command failed due to The broker only supports OffsetFetchRequest v1, but we need v2 or newer to request all topic partitions.

Seems like version incompatibility here to execute commands with 1.0 binaries.

yasha-podeswa-hs commented Apr 25, 2018

Anyone know if there’s a way to do this on Kafka 0.9.0 ?

dkurzaj commented Jul 31, 2018

Apparently my latests and my earliests offsets have the same values, and are equal to LOG-END-OFFSET. I can not manage to go back to the beginning of the messages.

mduhan commented Aug 18, 2018

kafka .10 does not support —rest-offsets but this can be achieved using java apis.
Please have a look.

manigeeth-narendula commented Apr 1, 2019

Can anyone tell me how to do this for kafka 1.0.0

OneCricketeer commented Apr 22, 2019

The API is all the same since 0.11. This includes 1.0.0, 2.0, and above until anyone mentions otherwise.

yuranos commented Jul 15, 2019

Applying 1.0 to 0.10 didn’t work for me either. Same as @Joginder22:
Error: Executing consumer group command failed due to The broker only supports OffsetFetchRequest v1, but we need v2 or newer to request all topic partitions.

Strech commented Nov 13, 2019 •

I would like to make a small adjustment about how to reset a topic partition

qinzl1 commented Nov 21, 2019

kafka1.1 doesn’t seem to support reset by —to-datetime. it can effect —to-earliest but do not effect —to-datetime

zwu01 commented Dec 10, 2019

Anyone knows if it’s possible to reset the group offset using librdkafkacpp APIs? Thanks

muscovitebob commented Feb 6, 2020

Just leaving a note here to save everyone using a Kafka cluster protected by SSL time. In case you are getting the following OOM error when running kafka-consumer-groups :

Counterintuitively, you are not running out of memory, but rather this arises as a result of not providing SSL credentials. You need to create a .properties file and pass it into each call to the binary via the —command-config option. In our case, the template for this properties file is:

Hope this saves someone time in the future.

Источник

ERROR Uncaught exception in herder work thread in Distributed mode #189

Comments

aldobd commented Feb 1, 2017 •

I am trying to start Kafka connect (Confluent 3.1.1) in distributed mode but I am having some problems:

[2017-02-01 11:08:52,803] ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:181)
java.lang.NullPointerException
at org.apache.kafka.connect.storage.KafkaConfigBackingStore$ConsumeCallback.onCompletion(KafkaConfigBackingStore.java:444)
[2017-02-01 11:08:51,893] INFO Discovered coordinator inaki-P552LA:9092 (id: 2147483647 rack: null) for group test. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:555)
[2017-02-01 11:08:52,803] ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:181)
java.lang.NullPointerException
at org.apache.kafka.connect.storage.KafkaConfigBackingStore$ConsumeCallback.onCompletion(KafkaConfigBackingStore.java:444)
at org.apache.kafka.connect.storage.KafkaConfigBackingStore$ConsumeCallback.onCompletion(KafkaConfigBackingStore.java:424)
at org.apache.kafka.connect.util.KafkaBasedLog.poll(KafkaBasedLog.java:253)
at org.apache.kafka.connect.util.KafkaBasedLog.readToLogEnd(KafkaBasedLog.java:293)
at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:143)
at org.apache.kafka.connect.storage.KafkaConfigBackingStore.start(KafkaConfigBackingStore.java:259)
at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:114)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:169)
at java.lang.Thread.run(Thread.java:745)
[2017-02-01 11:08:52,804] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:68)
[2017-02-01 11:08:52,804] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:154)
[2017-02-01 11:08:52,807] INFO Stopped ServerConnector@e3cee7b <0.0.0.0:8083>(org.eclipse.jetty.server.ServerConnector:306)
[2017-02-01 11:08:52,813] INFO Stopped o.e.j.s.ServletContextHandler@120f38e6 (org.eclipse.jetty.server.handler.ContextHandler:865)

I did some steps mentioned and If I start workers before connect it works without NullPointerException, but when I send messages throw timeout errors

The text was updated successfully, but these errors were encountered:

ewencp commented Feb 3, 2017

What version are you on and did you resolve this? We did used to see this sometimes and I think we tried to add some logging to give more info if it happened again because it was difficult to tell how things had gotten out of sync from looking at the code.

aldobd commented Feb 23, 2017

I could’t resolve it, I just restart everything to default following documentation and test again. So, the error disappeared

tuxedobitmask commented Nov 2, 2018 •

Reproducing this consistently with Kafka Connect 1.1.0 against a 1.0.0 cluster. Repro steps are:

1. Shutdown Kafka connect
2. Delete Kafka Connect topics (offsets, status, configs)
3. (assuming auto-creation enabled in cluster) Start Kafka Connect back up
4. Send GET request to https://my-kafka-connect/connectors

Symptoms, you’ll get a timeout on the GET request and an exception like this:

Interestingly, it will succeed in creating the offsets topic, but not the config or status topics.

Resetting cluster repeatedly (by deleting Kafka Connect topics) does not resolve.

This happened immediately after upgrading Kafka Connect from 1.0 to 1.1 . No other changes to cluster or deployment environment. @ewencp

tuxedobitmask commented Nov 2, 2018

I may have just figured this out. it looks like the status topic has been eternally marked for deletion (confirming from within ZK). It’s no longer listed in the broker metadata as an existing topic but it appears ZK never completed the delete. I think this causing a strange state where when Kafka Connect then tries to recreate it, Kafka can’t service the request because ZK disagrees. not sure if this is the same root cause as the original reported issue but it’s something.

tuxedobitmask commented Nov 2, 2018 •

We ended up forcing the controllers to reset their «deleting state» of the Kafka Connect topics by:

1. Shutdown Kafka connect
2. Running deletes from kafka-topics.sh
3. Deleting ZK entries
4. Creating topics (will appear to fail if controllers are still in «deleting» state but that’s ok)
5. Run deletes again from kafka-topics.sh (ensures resetting controllers «deleting» state)
6. Restart Kafka Connect to automatically create topics for real

gmanolache commented Oct 2, 2019

Had the same bug, kafka-connect was creating the 3 partitions connect-configs , connect-offsets and connect-status with the default number of partitions (in my case 64 as defined in the kafka server properties) and was causing this issue.

As the above solution, I ended up removing the topics, recreating them with 1 partition and restarting kafka-connect.

liranshaked commented Dec 25, 2019 •

I am unable to start kafka connect
See in the kafka connect logs :

Источник

KAFKA и SSL: java.lang.OutOfMemoryError: пространство кучи Java при использовании команды kafka-themes в кластере KAFKA SSL

это мой первый пост в Stackoverflow, надеюсь, я не ошибся.

Размер Kafka HEAP настраивается в следующем файле:

Со следующим параметром:

ОС — «CentOS Linux, выпуск 7.7.1908» .

Kafka — это «confluent-kafka-2.12-5.3.1-1.noarch» , установленный из следующего репозитория:

Несколько дней назад я активировал SSL на кластере KAFKA из трех машин, и внезапно перестала работать следующая команда:

Которые возвращают мне следующую ошибку:

В журнале сервера появляется следующая строка, когда я пытаюсь запросить ее через «kafka-themes»:

Я смог правильно использовать эту команду ДО внедрения SSL в кластере. Вот конфигурация, которую я использую. Все функции работают исправно (потребители, производители . ) кроме «кафка-топиков»:

Нет проблем с сертификатом (который подписан внутренним ЦС, внутренним ЦС, который я добавил в хранилище доверенных сертификатов, указанное в конфигурации). OpenSSL не показывает ошибок:

Следующая команда отлично работает с SSL благодаря параметру «-consumer.config client-ssl.properties»

Прямо сейчас я вынужден использовать «—zookeeper», который, согласно документации, устарел:

И, конечно, неплохо работает:

Итак, мой вопрос: почему я не могу использовать банкомат «—bootstrap-server»? Из-за прекращения поддержки «zookeeper» я беспокоюсь, что не смогу ознакомиться с моими темами и их деталями .

Я считаю, что для kafka-themes нужен тот же параметр, что и для kafka-console-consumer, он же «-consumer.config» .

Спросите, нужна ли дополнительная точность.

Большое спасибо, надеюсь, мой вопрос ясен и удобочитаем.

Источник

Это мой первый пост на Stackoverflow, надеюсь, я выбрал не тот раздел.

Контекст:

Размер Kafka HEAP настраивается для следующего файла:

/etc/systemd/system/kafka.service

Со следующим параметром:

Environment="KAFKA_HEAP_OPTS=-Xms6g -Xmx6g"

ОС «CentOS Linux выпуск 7.7.1908» .

Kafka — это «confluent-kafka-2.12-5.3.1-1.noarch» , установленный из следующего репозитория:

# Confluent REPO
[Confluent.dist]
name=Confluent repository (dist)
baseurl=http://packages.confluent.io/rpm/5.3/7
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/5.3/archive.key
enabled=1

[Confluent]
name=Confluent repository
baseurl=http://packages.confluent.io/rpm/5.3
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/5.3/archive.key
enabled=1

Я активировал SSL на 3-машинном кластере KAFKA несколько дней назад, и вдруг следующая команда перестала работать:

kafka-topics --bootstrap-server <the.fqdn.of.server>:9093 --describe --topic <TOPIC-NAME>

Которые возвращают мне следующую ошибку:

[2019-10-03 11:38:52,790] ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1':(org.apache.kafka.common.utils.KafkaThread) 
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
    at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1152)
    at java.lang.Thread.run(Thread.java:748)

В журнале сервера появляется следующая строка, когда я пытаюсь запросить его через «kafka-themes»:

/var/log/kafka/server.log :
[2019-10-03 11:41:11,913] INFO [SocketServer brokerId=<ID>] Failed authentication with /<ip.of.the.server> (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Я смог правильно использовать эту команду ДО внедрения SSL в кластере. Вот конфигурация, которую я использую. Все функции работают должным образом (потребители, производители …), кроме «Кафка-топики»:

# SSL Configuration
ssl.truststore.location=<truststore-path>
ssl.truststore.password=<truststore-password>
ssl.keystore.type=<keystore-type>
ssl.keystore.location=<keystore-path>
ssl.keystore.password=<keystore-password>

# Enable SSL between brokers
security.inter.broker.protocol=SSL

# Listeners
listeners=SSL://<fqdn.of.the.server>:9093
advertised.listeners=SSL://<fqdn.of.the.server>:9093

Нет проблем с сертификатом (который подписан внутренним CA, внутренним CA, который я добавил в склад доверенных сертификатов, указанный в конфигурации). OpenSSL не показывает ошибок:

openssl s_client -connect <fqdn.of.the.server>:9093 -tls1
>> Verify return code: 0 (ok)

Следующая команда очень хорошо работает с SSL, благодаря параметру «-consumer.config client-ssl.properties»

kafka-console-consumer --bootstrap-server <fqdn.of.the.server>:9093 --topic <TOPIC-NAME> -consumer.config client-ssl.properties

Содержимое «client-ssl.properties»:

security.protocol=SSL
ssl.truststore.location=<truststore-path>
ssl.truststore.password=<truststore-password>

Сейчас я вынужден использовать «—zookeeper», который, согласно документации, устарел:

--zookeeper <String: hosts>              DEPRECATED, The connection string for  
                                       the zookeeper connection in the form 
                                       host:port. Multiple hosts can be     
                                       given to allow fail-over. 

И, конечно, это работает довольно хорошо:

kafka-topics --zookeeper <fqdn.of.the.server>:2181 --describe --topic <TOPIC-NAME>
Topic:<TOPIC-NAME>  PartitionCount:3    ReplicationFactor:2 
Configs:
Topic: <TOPIC-NAME> Partition: 0    Leader: <ID-3>      Replicas: <ID-3>,<ID-1> Tsr: <ID-1>,<ID-3>
Topic: <TOPIC-NAME> Partition: 1    Leader: <ID-1>      Replicas: <ID-1>,<ID-2> Isr: <ID-2>,<ID-1>
Topic: <TOPIC-NAME> Partition: 2    Leader: <ID-2>      Replicas: <ID-2>,<ID-3> Isr: <ID-2>,<ID-3>

Итак, мой вопрос: почему я не могу использовать «—bootstrap-server» банкомат? Из-за устаревшего «zookeeper» я беспокоюсь, что не смогу проконсультироваться с моими темами и их деталями …

Я считаю, что для kafka-themes нужна та же опция, что и для kafka-console-consumer, или «-consumer.config» …

Спросите, нужна ли дополнительная точность.

Большое спасибо, надеюсь, мой вопрос ясен и читаем.

Blyyyn

Я наконец нашел способ справиться с этой ошибкой SSL. Ключ должен использовать следующие настройки:

--command-config client-ssl.properties

Это работает с большей частью команд KAFKA, таких как kafka-consumer-groups и, конечно, kafka-themes . Смотрите примеры ниже:

kafka-consumer-groups --bootstrap-server <kafka-hostname>:<kafka-port> --group <consumer-group> --topic <topic> --reset-offsets --to-offset <offset> --execute --command-config <ssl-config>

kafka-topics --list --bootstrap-server <kafka-hostname>:<kafka-port> --command-config client-ssl.properties

ssl-config был «client-ssl.properties», см. исходное сообщение для содержимого. Помните, что при использовании IP-адреса вы получите ошибку, если у сертификата машины нет альтернативного имени с этим IP-адресом. Попробуйте иметь правильное разрешение DNS и используйте полное доменное имя, если это возможно.

Надеюсь, что это решение поможет, ура!

Blyyyn