Logstash attempted to resurrect connection to dead es instance but got an error - Исправление ошибок и поиск оптимальных решений проблем

I’ve also got the same problem.

[2021-06-27T15:35:30,864][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://locahost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://locahost:9200/][Manticore::ResolutionFailure] locahost"}

Whereas elasticsearch is available-

curl http://localhost:9200

{
  "name" : "ELK",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "i90k-uQlQjyERkduOLy6Jw",
  "version" : {
    "number" : "7.13.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "4d960a0733be83dd2543ca018aa4ddc42e956800",
    "build_date" : "2021-06-10T21:01:55.251515791Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Strangely when I try to stop logstash service, I get this problem from logstash-plain.log

[2021-06-27T15:38:21,684][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>32, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}, {"thread_id"=>33, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}, {"thread_id"=>34, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}, {"thread_id"=>35, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}]}}

Then I’ve to kill the logstash process. Although, I am able to upload a csv using a custom config file using below command-
/usr/share/logstash/bin/logstash -f paloAlto.config
Plz let me know if more logs are required.

Источник

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Источник

Hii i tried your method now I am getting all sorts of errors in kibana logstash. I am unable to resolve them

from logstash_async.formatter import LogstashFormatter
from logstash_async.handler import AsynchronousLogstashHandler

host = '10.20.210.40'
port = 5044



class Logger:
    def __init__(self, service):
        self.service = service
        logstash_formatter = LogstashFormatter(
            extra={
                'application': 'sizzle',
                'custom_log_version': '0.9.9',
                'environment': 'production',
                'microservice': service,
            })
        logstash_handler = AsynchronousLogstashHandler(
            host,
            port,
            database_path='/tmp/logstash-{}.db'.format(os.getpid())
        )

        logstash_handler.setFormatter(logstash_formatter)
        self.logger = logging.getLogger('python-logstash-logger')
        self.logger.setLevel(logging.INFO)
        self.logger.addHandler(logstash_handler)

    def _dynamic_logging(self, level, log_message, queue_record, extra):
        if level in ['info', 'debug', 'warning', 'error', 'critical']:
            extra_fields = {}

            if queue_record is not None:
                extra_fields['message_id'] = queue_record['_id']
                if 'streamer' in queue_record:
                    extra_fields['streamer_name'] = queue_record['streamer']['name']
                    extra_fields['streamer_id'] = queue_record['streamer']['_id']
                if 'twitch' in queue_record:
                    extra_fields['twitch_stream_id'] = queue_record['twitch']['id']
                    extra_fields['twitch_stream_published_at'] = queue_record['twitch']['published_at']
                if 'video_file' in queue_record:
                    if 'duration_seconds' in queue_record['video_file']:
                        extra_fields['video_duration'] = queue_record['video_file']['duration_seconds']
                    if 'frames' in queue_record['video_file']:
                        extra_fields['video_number_of_frames'] = queue_record['video_file']['frames']
            if extra is not None:
                extra_fields.update(extra)

            # print("Log Message: {}, extras: {}".format(log_message, extra_fields))
            getattr(self.logger, level)(log_message, extra=extra_fields)

    def info(self, log_message, queue_record=None, extra=None):
        self._dynamic_logging('info', log_message=log_message, queue_record=queue_record, extra=extra)

    def debug(self, log_message, queue_record=None, extra=None):
        self._dynamic_logging('debug', log_message=log_message, queue_record=queue_record, extra=extra)

    def warning(self, log_message, queue_record=None, extra=None):
        self._dynamic_logging('warning', log_message=log_message, queue_record=queue_record, extra=extra)

    def error(self, log_message, queue_record=None, extra=None):
        self._dynamic_logging('error', log_message=log_message, queue_record=queue_record, extra=extra)

    def critical(self, log_message, queue_record=None, extra=None):
        self._dynamic_logging('critical', log_message=log_message, queue_record=queue_record, extra=extra)

Источник

UP 01.07.2020

Здравствуйте, помогите разобраться с ошибкой

systemctl restart logstash.service &&  tail -f /var/log/logstash/logstash-plain.log
[2020-07-01T09:02:20,251][INFO ][logstash.runner          ] Logstash shut down.
[2020-07-01T09:02:36,912][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.8.0", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Server VM 25.252-b09 on 1.8.0_252-8u252-b09-1~deb9u1-b09 +indy +jit [linux-x86_64]"}
[2020-07-01T09:02:37,832][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 1, column 1 (byte 1)", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}
[2020-07-01T09:02:38,272][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-07-01T09:02:40,270][WARN ][logstash.runner          ] SIGTERM received. Shutting down.
[2020-07-01T09:02:43,149][INFO ][logstash.runner          ] Logstash shut down.
[2020-07-01T09:02:59,563][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.8.0", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Server VM 25.252-b09 on 1.8.0_252-8u252-b09-1~deb9u1-b09 +indy +jit [linux-x86_64]"}
[2020-07-01T09:03:00,510][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 1, column 1 (byte 1)", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}
[2020-07-01T09:03:00,900][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-07-01T09:03:05,836][INFO ][logstash.runner          ] Logstash shut down.

Стоит EKL — так же стоит NGINX — на котором настроена авторизация.

Задача собрать лог с микротика, для етого в conf.d есть три файла

input.conf

input {
    beats {
	port => 5044
    }
    syslog {
	port => 5045
	type => syslog
    }
}

filter.conf

else if [host] == "10.1.4.19" or [host] == "10.1.5.1" {
        mutate {
            add_tag => [ "mikrotik", "gateway" ]
        }
    }
    else if [host] == "10.1.4.66" or [host] == "10.1.3.110" or [host] == "10.1.3.111" {
        mutate {
            add_tag => [ "mikrotik", "wifi" ]
        }
    }
    else if [host] == "10.1.4.14" or [host] == "10.1.5.33" {
        mutate {
            add_tag => [ "mikrotik", "switch" ]
	}
    }

output.conf

else if "mikrotik" in [tags] {
        elasticsearch {
            hosts     => "localhost:9200"
            index    => "mikrotik-%{+YYYY.MM}"
        }
    }

Елестик — запущен

curl -X GET http://localhost:9200
{
  "name" : "log",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "kUdSHXQsS3itWMHC8eFMKw",
  "version" : {
    "number" : "7.8.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "757314695644ea9a1dc2fecd26d1a43856725e65",
    "build_date" : "2020-06-14T19:35:50.234439Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"

но при этом получаю ошибку выше

куда копать?

Источник

Recommend Projects

<img decoding="async" onError="javascript: wp_broken_images = window.wp_broken_images || function(){}; wp_broken_images(this);" loading="lazy" width="24" height="24" src="https://www.typescriptlang.org/favicon-32x32.png" alt="Typescript photo" /> Typescript

<img decoding="async" onError="javascript: wp_broken_images = window.wp_broken_images || function(){}; wp_broken_images(this);" loading="lazy" width="24" height="24" src="https://avatars2.githubusercontent.com/u/27804?s=200&amp;v=4" alt="Django photo" /> Django

<img decoding="async" onError="javascript: wp_broken_images = window.wp_broken_images || function(){}; wp_broken_images(this);" loading="lazy" width="24" height="24" src="https://laravel.com/img/logomark.min.svg" alt="Laravel photo" /> Laravel

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

<img decoding="async" onError="javascript: wp_broken_images = window.wp_broken_images || function(){}; wp_broken_images(this);" loading="lazy" width="24" height="24" src="https://avatars.githubusercontent.com/u/69631?v=4" alt="Facebook photo" /> Facebook

<img decoding="async" onError="javascript: wp_broken_images = window.wp_broken_images || function(){}; wp_broken_images(this);" loading="lazy" width="24" height="24" src="https://avatars.githubusercontent.com/u/6154722?v=4" alt="Microsoft photo" /> Microsoft

<img decoding="async" onError="javascript: wp_broken_images = window.wp_broken_images || function(){}; wp_broken_images(this);" loading="lazy" width="24" height="24" src="https://avatars.githubusercontent.com/u/1961952?v=4" alt="Alibaba photo" /> Alibaba

<img decoding="async" onError="javascript: wp_broken_images = window.wp_broken_images || function(){}; wp_broken_images(this);" loading="lazy" width="24" height="24" src="https://avatars.githubusercontent.com/u/1562726?v=4" alt="D3 photo" /> D3

<img decoding="async" onError="javascript: wp_broken_images = window.wp_broken_images || function(){}; wp_broken_images(this);" loading="lazy" width="24" height="24" src="https://avatars.githubusercontent.com/u/18461506?v=4" alt="Tencent photo" /> Tencent

Читайте также:

Typescript

Django

Laravel

Facebook

Microsoft

Alibaba

D3

Tencent