Error writing to outputs influxdb could not write any address

@danielnelson sorry for the long wait, here are the info about my setup and configurations.

Current configuration & setup

The setup

• several Telegraf clients that monitor one or more SQL Server instances
• The data are sent to a telegraf gateway and redirected
• The gateway sends data to different InfluxDBs (using database_tag)

Client configuration

Version: Telegraf unknown (git: master 1b1382c)
The configuration is split into several files, for full detail see attachment
Configuration.zip

Server configuration (gateway)

Version: Telegraf 1.14.2 (git: HEAD fb74eaf)
there are 2 outputs only because some data are cloned in a specific DB
gateway.zip

Log since last Telegraf restart

2020-07-13T09:37:48Z I! Loaded inputs: internal sqlserver sqlserver
2020-07-13T09:37:48Z I! Loaded aggregators: 
2020-07-13T09:37:48Z I! Loaded processors: converter strings
2020-07-13T09:37:48Z I! Loaded outputs: influxdb
2020-07-13T09:37:48Z I! Tags enabled: company=quantumdatis_demo host=QDSRVMONITOR
2020-07-13T09:37:48Z I! [agent] Config: Interval:15s, Quiet:false, Hostname:"QDSRVMONITOR", Flush Interval:10s
2020-07-13T09:38:06Z E! [inputs.sqlserver] Error in plugin: Unable to get instances from Sql Server Browser on host SQLCSRV04: read udp 10.0.1.122:50483->10.0.1.110:1434: i/o timeout
2020-07-13T09:47:00Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T10:01:30Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T10:02:00Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T10:11:40Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T10:11:40Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T10:12:40Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T10:12:40Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T10:13:10Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T10:13:10Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T10:14:10Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T10:14:10Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T10:15:10Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T10:15:10Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T10:20:20Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T10:20:20Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T10:23:20Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T10:23:20Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T10:55:30Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T11:00:00Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T11:52:45Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T11:54:25Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T11:54:25Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T11:54:55Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T11:54:55Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T11:57:55Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T11:57:55Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T11:58:25Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T11:58:25Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T12:00:00Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T12:00:00Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T12:02:05Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T12:02:05Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T12:02:55Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T12:02:55Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T12:04:05Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T12:04:05Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:26:00Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T13:30:00Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T13:33:49Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:33:49Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:43:44Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020-07-13T13:43:44Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:46:10Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:46:10Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:46:40Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:46:40Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:47:10Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:47:10Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:48:10Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:48:10Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:48:40Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:48:40Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:49:30Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:49:30Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:50:40Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:50:40Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:51:20Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:51:20Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:53:50Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:53:50Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T13:54:20Z E! [outputs.influxdb] When writing to [https://_MyUrl_]: Post "https://_MyUrl_/write?db=quantumdatis_demo": EOF
2020-07-13T13:54:20Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-07-13T14:00:00Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T14:30:00Z W! [agent] [inputs.sqlserver] did not complete within its interval
2020-07-13T15:00:00Z W! [agent] [inputs.sqlserver] did not complete within its interval

I’ve noticed that the Telegraf gateway has an «old» version, I could start by updating it and check if the error persists.

If you have something to try to get more data about the issue just let me know and I will configure/apply it as soon as possible

This topic has been deleted. Only users with topic management privileges can see it.

• I am trying to get Telegraf to send data to an InfluxDB instance. It was working until I rebooted the firewall, and now I can’t make it work no matter what I try. I verified that telegraf is running, and also double checked telegraf.conf for the correct IP of the InfluxDB instance. It just doesnt seem to be sending any data out of pfsense. My next step will be to run wireshark on the network to see where the packets are stopping.

  Any other ideas?

• Check the log file.

  
  tail -f /var/log/telegraf.log
  2017/02/25 13:25:50 E! Database creation failed: Post http://192.168.86.30:8086/query?db=&q=CREATE+DATABASE+%22telegraf%22: dial tcp 192.168.86.30:8086: connect: host is down
  2017/02/25 13:25:50 E! Error writing to output [influxdb]: Could not write to any InfluxDB server in cluster
  2017/02/25 13:26:00 E! Database creation failed: Post http://192.168.86.30:8086/query?db=&q=CREATE+DATABASE+%22telegraf%22: dial tcp 192.168.86.30:8086: connect: host is down
  
  

  When I bounce my influxdb, I get error messages in the log.

  Most likely, you have a config issue and it should show up in the telegraf log.

• I just checked the telegraf log, and below is the repeating log entry.  I verified the IP address of the InfluxDB server, and I am not using a username/password yet.

  2017/02/21 20:33:38 E! Database creation failed: Post http://localhost:8086/query?db=&q=CREATE+DATABASE+%22telegraf%22: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  2017/02/21 20:33:55 E! Database creation failed: Post http://localhost:8086/query?db=&q=CREATE+DATABASE+%22telegraf%22: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  2017/02/21 20:33:55 E! Error writing to output [influxdb]: Could not write to any InfluxDB server in cluster
  

  The telegraf.conf seems fairly straightforward for at least basic use, so I  am pretty sure it is ok. On another note, I had this working last week, and was graphing simple data in Grafana, until I rebooted pfsense, then all data flow stopped. I cant figure out what would have changed upon reboot to break this.

• It looks like you have the default localhost config based on the error.

  What does your urls line look like:

  
    # urls = ["udp://localhost:8089"] # UDP endpoint example
    urls = ["http://192.168.86.30:8086"] # required
  
  

  Mine writes to the 192 address above where my influxdb is.

• I may have posted the wrong log. I have been testing this various ways, and I think I wasnt thinking clearly.  ::) I have been trying everything I can think of to remedy this. Below is the current config and resulting log entries.

  [[outputs.influxdb]]
    ## The full HTTP or UDP endpoint URL for your InfluxDB instance.
    ## Multiple urls can be specified as part of the same cluster,
    ## this means that only ONE of the urls will be written to each interval.
    # urls = ["udp://10.20.0.232:8089"] # UDP endpoint example
    urls = ["http://10.20.0.55:8086"] # required
    ## The target database for metrics (telegraf will create it if not exists).
    database = "telegraf" # required
  
  

  2017/02/28 17:09:45 E! Database creation failed: Post http://10.20.0.55:8086/query?db=&q=CREATE+DATABASE+%22telegraf%22: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  2017/02/28 17:09:45 E! Error writing to output [influxdb]: Could not write to any InfluxDB server in cluster
  

• Are you able to hit:

  ttp://10.20.0.55:8083

  That’s normally your admin port.

  You can enter in ‘show databases’

  and even try to create the database.

  I have no authentication on my influxdb since it’s all internal.

I previously installed Influxdb and telegraf on a windows2019 server for testing and it have worked well. Now I have installed a second influxdb in a OpenShift pod that uses https for communication. I have connected to the new database by using “InfluxDB Studio” and I can create new databases and add/edit data from there.

The only thing I had to do was to check the “SSL” box when creating the connection to the database.

But I can’t get telegraf to write any data to the new database. I edited the config-file and updated the url but no connection, I also tried altering “insecure_skip_verify” to true/false/comment but nothing changes.

This is the errors I get:

2021-01-19T14:41:45Z I! Loaded inputs: ping exec win_perf_counters
2021-01-19T14:41:45Z I! Loaded aggregators: 
2021-01-19T14:41:45Z I! Loaded processors: 
2021-01-19T14:41:45Z I! Loaded outputs: influxdb
2021-01-19T14:41:45Z I! Tags enabled: host=LML-CND9032X5B
2021-01-19T14:41:45Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"LML-CND9032X5B", Flush Interval:10s
2021-01-19T14:41:45Z D! [agent] Initializing plugins
2021-01-19T14:41:45Z D! [agent] Connecting outputs
2021-01-19T14:41:45Z D! [agent] Attempting connection to [outputs.influxdb]
2021-01-19T14:41:45Z W! [outputs.influxdb] When writing to [https://demo-app00013-influxdb-test.caas-internal.lm.se:443]: database "telegraf" creation failed: 400 Bad Request
2021-01-19T14:41:45Z D! [agent] Successfully connected to outputs.influxdb
2021-01-19T14:41:45Z D! [agent] Starting service inputs
2021-01-19T14:42:00Z E! [outputs.influxdb] When writing to [https://demo-app00013-influxdb-test.caas-internal.lm.se:443]: 400 Bad Request
2021-01-19T14:42:00Z D! [outputs.influxdb] Buffer fullness: 136 / 10000 metrics
2021-01-19T14:42:00Z E! [agent] Error writing to outputs.influxdb: could not write any address

config:

# Telegraf Configuration
#
# Telegraf is entirely plugin driven. All metrics are gathered from the
# declared inputs, and sent to the declared outputs.
#
# Plugins must be declared in here to be active.
# To deactivate a plugin, comment out the name and any variables.
#
# Use 'telegraf -config telegraf.conf -test' to see what metrics a config
# file would generate.
#
# Environment variables can be used anywhere in this config file, simply surround
# them with ${}. For strings the variable must be within quotes (ie, "${STR_VAR}"),
# for numbers and booleans they should be plain (ie, ${INT_VAR}, ${BOOL_VAR})


# Global tags can be specified here in key="value" format.
[global_tags]
  # dc = "us-east-1" # will tag all metrics with dc=us-east-1
  # rack = "1a"
  ## Environment variables can be used as tags, and throughout the config file
  # user = "$USER"


# Configuration for telegraf agent
[agent]
  ## Default data collection interval for all inputs
  interval = "10s"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  ## Telegraf will send metrics to outputs in batches of at most
  ## metric_batch_size metrics.
  ## This controls the size of writes that Telegraf sends to output plugins.
  metric_batch_size = 1000

  ## Maximum number of unwritten metrics per output.  Increasing this value
  ## allows for longer periods of output downtime without dropping metrics at the
  ## cost of higher maximum memory usage.
  metric_buffer_limit = 10000

  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"

  ## Default flushing interval for all outputs. Maximum flush_interval will be
  ## flush_interval + flush_jitter
  flush_interval = "10s"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"

  ## By default or when set to "0s", precision will be set to the same
  ## timestamp order as the collection interval, with the maximum being 1s.
  ##   ie, when interval = "10s", precision will be "1s"
  ##       when interval = "250ms", precision will be "1ms"
  ## Precision will NOT be used for service inputs. It is up to each individual
  ## service input to set the timestamp at the appropriate precision.
  ## Valid time units are "ns", "us" (or "µs"), "ms", "s".
  precision = ""

  ## Log at debug level.
  debug = true
  ## Log only error level messages.
  quiet = false

  ## Log target controls the destination for logs and can be one of "file",
  ## "stderr" or, on Windows, "eventlog".  When set to "file", the output file
  ## is determined by the "logfile" setting.
  logtarget = "file"

  ## Name of the file to be logged to when using the "file" logtarget.  If set to
  ## the empty string then logs are written to stderr.
  # logfile = "C:/Program Files/Telegraf"
  logfile = "/Temp/telegraf.log"

  ## The logfile will be rotated after the time interval specified.  When set
  ## to 0 no time based rotation is performed.  Logs are rotated only when
  ## written to, if there is no log activity rotation may be delayed.
  # logfile_rotation_interval = "0d"

  ## The logfile will be rotated when it becomes larger than the specified
  ## size.  When set to 0 no size based rotation is performed.
  # logfile_rotation_max_size = "0MB"

  ## Maximum number of rotated archives to keep, any older logs are deleted.
  ## If set to -1, no archives are removed.
  # logfile_rotation_max_archives = 5

  ## Override default hostname, if empty use os.Hostname()
  hostname = ""
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = false


###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################


# Configuration for sending metrics to InfluxDB
[[outputs.influxdb]]
  ## The full HTTP or UDP URL for your InfluxDB instance.
  ##
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.
  # urls = ["unix:///var/run/influxdb.sock"]
  # urls = ["udp://127.0.0.1:8089"]
   urls = ["https://demo-app00013-influxdb-test.caas-internal.lm.se:443"]
  username = "root"
  password = "root"
  ## The target database for metrics; will be created as needed.
  ## For UDP url endpoint database needs to be configured on server side.
   database = "telegraf"
   #database = "test"

  ## The value of this tag will be used to determine the database.  If this
  ## tag is not set the 'database' option is used as the default.
  # database_tag = ""

  ## If true, the database tag will not be added to the metric.
  # exclude_database_tag = false

  ## If true, no CREATE DATABASE queries will be sent.  Set to true when using
  ## Telegraf with a user without permissions to create databases or when the
  ## database already exists.
   skip_database_creation = false

  ## Name of existing retention policy to write to.  Empty string writes to
  ## the default retention policy.  Only takes effect when using HTTP.
  # retention_policy = ""

  ## Write consistency (clusters only), can be: "any", "one", "quorum", "all".
  ## Only takes effect when using HTTP.
  # write_consistency = "any"

  ## Timeout for HTTP messages.
  # timeout = "5s"

  ## HTTP Basic Auth 
  # username = "telegraf"
  # password = "metricsmetricsmetricsmetrics"

  ## HTTP User-Agent
  user_agent = "telegraf"

  ## UDP payload size is the maximum packet size to send.
  # udp_payload = "512B"

  ## Optional TLS Config for use on HTTP connections.
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  insecure_skip_verify = false

  ## HTTP Proxy override, if unset values the standard proxy environment
  ## variables are consulted to determine which proxy, if any, should be used.
  # http_proxy = "http://corporate.proxy:3128"

  ## Additional HTTP headers
  # http_headers = {"X-Special-Header" = "Special-Value"}

  ## HTTP Content-Encoding for write request body, can be set to "gzip" to
  ## compress body or "identity" to apply no encoding.
   #content_encoding = "text/plain; charset=UTF-8"

  ## When true, Telegraf will output unsigned integers as unsigned values,
  ## i.e.: "42u".  You will need a version of InfluxDB supporting unsigned
  ## integer values.  Enabling this option will result in field type errors if
  ## existing data has been written.
  # influx_uint_support = false

# # Configuration for sending metrics to InfluxDB
# [[outputs.influxdb_v2]]
#   ## The URLs of the InfluxDB cluster nodes.
#   ##
#   ## Multiple URLs can be specified for a single cluster, only ONE of the
#   ## urls will be written to each interval.
#   ##   ex: urls = ["https://us-west-2-1.aws.cloud2.influxdata.com"]
#   urls = ["http://127.0.0.1:9999"]
#
#   ## Token for authentication.
#   token = ""
#
#   ## Organization is the name of the organization you wish to write to; must exist.
#   organization = ""
#
#   ## Destination bucket to write into.
#   bucket = ""
#
#   ## The value of this tag will be used to determine the bucket.  If this
#   ## tag is not set the 'bucket' option is used as the default.
#   # bucket_tag = ""
#
#   ## If true, the bucket tag will not be added to the metric.
#   # exclude_bucket_tag = false
#
#   ## Timeout for HTTP messages.
#   # timeout = "5s"
#
#   ## Additional HTTP headers
#   # http_headers = {"X-Special-Header" = "Special-Value"}
#
#   ## HTTP Proxy override, if unset values the standard proxy environment
#   ## variables are consulted to determine which proxy, if any, should be used.
#   # http_proxy = "http://corporate.proxy:3128"
#
#   ## HTTP User-Agent
#   # user_agent = "telegraf"
#
#   ## Content-Encoding for write request body, can be set to "gzip" to
#   ## compress body or "identity" to apply no encoding.
#    content_encoding = "text/html; charset=UTF-8"
#
#   ## Enable or disable uint support for writing uints influxdb 2.0.
#   # influx_uint_support = false
#
#   ## Optional TLS Config for use on HTTP connections.
#   # tls_ca = "/etc/telegraf/ca.pem"
#   # tls_cert = "/etc/telegraf/cert.pem"
#   # tls_key = "/etc/telegraf/key.pem"
#   ## Use TLS but skip chain & host verification
#   # insecure_skip_verify = false