Error on ingesting samples with different value but same timestamp

What did you do?
Upgraded to 1.5.2 and run it for some time.

Environment and logs
Dockerized Prometheus node with 256GB of RAM and 12cores, monitoring around 1.5k nodes, at least 3 endpoints on each host.

System information:

# uname -srm
Linux 4.4.0-47-generic x86_64

Prometheus version:

prometheus, version 1.5.2 (branch: master, revision: bd1182d29f462c39544f94cc822830e1c64cf55b)
  build user:       root@1a01c5f68840
  build date:       20170210-16:23:28
  go version:       go1.7.5

Alertmanager version:

alertmanager, version 0.5.1 (branch: master, revision: 0ea1cac51e6a620ec09d053f0484b97932b5c902)
  build user:       root@fb407787b8bf
  build date:       20161125-08:14:40
  go version:       go1.7.3

Prometheus configuration file:

# my global config
global:
  scrape_interval:     30s # By default, scrape targets every 15 seconds.
  evaluation_interval: 30s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'bhss'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  - "recording.rules"
  - "alert.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  # Auto-discovery via consul tags by Registrator
  - job_name: 'consul-services'

    consul_sd_configs:
    - server: 'consul:8500'
      scheme: 'http'


    metric_relabel_configs:
      - source_labels: [__name__]
        regex: go_(.*)
        action: drop
      - source_labels: [__name__]
        regex: container_(fs|network|spec)_(.*)
        action: drop
      - source_labels: [__name__]
        regex: http_(.*)
        action: drop
      - source_labels: [__name__]
        regex: node_(net|sock|vm)stat_(.*)
        action: drop
      - source_labels: [__name__]
        regex: prometheus_(.*)
        action: drop
      - source_labels: [__name__]
        regex: docker_container_(blkio|net)(.*)
        action: drop
      - source_labels: [__name__]
        regex: net_(icmp|tcp|udp|ip)(.*)
        action: drop
      - source_labels: [__name__]
        regex: nstat_(.*)
        action: drop
      - target_label: id
        replacement: ''
      - target_label: container_label_com_docker_compose_config_hash
        replacement: ''
      - target_label: container_label_build_date
        replacement: ''
      - target_label: container_label_com_docker_compose_container_number
        replacement: ''
      - target_label: container_label_com_docker_compose_oneoff
        replacement: ''
      - target_label: container_label_com_docker_compose_project
        replacement: ''
      - target_label: container_label_com_docker_compose_service
        replacement: ''
      - target_label: container_label_com_docker_compose_version
        replacement: ''
      - target_label: container_label_license
        replacement: ''
      - target_label: container_label_name
        replacement: ''
      - target_label: io_kubernetes_container_name
        replacement: ''
      - target_label: io_kubernetes_pod_name
        replacement: ''
      - target_label: io_kubernetes_container_restartCount
        replacement: ''

    relabel_configs:
    - source_labels: [ '__meta_consul_tags' ]
      action: keep
      regex: .*exporter.*
    - source_labels: [ '__meta_consul_address', '__meta_consul_service_port' ]
      action: replace
      regex: (.+)(?::d+);(d+)
      replacement: $1:$2
      target_label: __address__
    - source_labels: [ '__meta_consul_service' ]
      action: keep
      regex: (.+)
      replacement: $1
      target_label: __name__
    - source_labels: [ '__meta_consul_service_id' ]
      action: replace
      regex: (.*):(.*):(.*)
      replacement: $2
      target_label: container_name
    - source_labels: [ '__meta_consul_service_id' ]
      action: replace
      regex: (.*):(.*):(.*)
      replacement: $1
      target_label: host_name
    - source_labels: [ '__meta_consul_service_id' ]
      action: replace
      regex: .*-(w*)-(w*)-(d*):(.*):(.*)
      replacement: $1
      target_label: colo
    - source_labels: [ '__meta_consul_service_id' ]
      action: replace
      regex: (w*-w*-w*)-(d*):(.*):(.*)
      replacement: $1
      target_label: host_type

Logs:

ime="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1136 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1135 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1123 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1163 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1226 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1124 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1163 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1146 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1117 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1130 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1179 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1097 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1212 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1184 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1160 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1075 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1155 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1317 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1141 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1128 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1102 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1154 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1127 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1129 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1145 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1123 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1368 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1144 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1176 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1136 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1223 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1152 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1051 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1219 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1116 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1063 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1082 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1100 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1185 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1111 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1185 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1120 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1233 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1151 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1229 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1069 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1151 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1087 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1082 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1181 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1071 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1129 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1138 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1299 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1083 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1246 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1086 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1132 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1206 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1147 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1161 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1116 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1150 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1079 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1283 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1151 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1120 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1138 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1181 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1150 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1072 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1177 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1127 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1166 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1232 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1099 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1136 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1128 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1105 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1147 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1200 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1374 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1105 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1144 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1186 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1139 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1155 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1162 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1178 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1115 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1152 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1161 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1189 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1076 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1097 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1119 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1176 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1186 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1163 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1080 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1142 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1132 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1148 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1074 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1363 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1223 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1216 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1087 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1196 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1175 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1100 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1148 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1181 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1099 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1115 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1169 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1105 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1108 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1159 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1077 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1130 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1200 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1119 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1122 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1111 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1129 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1132 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1192 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1082 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1087 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1176 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1115 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1126 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1068 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1332 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1100 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1344 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1146 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1162 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1348 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1130 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1181 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1185 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1229 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1020 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1184 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1122 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1073 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1064 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1173 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1111 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1148 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1177 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1146 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1145 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1079 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1130 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1133 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1092 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1193 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1196 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1212 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1172 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1207 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1129 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1135 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1072 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1190 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1086 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1092 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1108 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1083 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1151 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1097 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1067 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1120 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1116 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1231 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1124 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1142 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1059 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1139 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1083 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1128 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1061 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1133 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1189 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:612"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1149 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1192 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1177 source="scrape.go:520"
time="2017-03-12T20:18:05Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1074 source="scrape.go:520"
time="2017-03-12T20:18:21Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=787 source="scrape.go:520"
time="2017-03-12T20:18:22Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1201 source="scrape.go:520"
time="2017-03-12T20:18:23Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:23Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1108 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1198 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1119 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1161 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1193 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1059 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1184 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1125 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1087 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1118 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1086 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1202 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1056 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1228 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1068 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1173 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1204 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1239 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1102 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=990 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1122 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1159 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1097 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1154 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1191 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1122 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1125 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1120 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1065 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1141 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1119 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1215 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1159 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1105 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1155 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1157 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1182 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1165 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1206 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1138 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1017 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1081 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1108 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1183 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1025 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1086 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1179 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1160 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1184 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1135 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1179 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1144 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1206 source="scrape.go:520"

Источник

Содержание

Prometheus has many samples rejected
Share this:
Need personalized help?
Error on ingesting samples with different value and the prometheus log does not show #2858
Comments
Error on ingesting samples with different value but same timestamp

Prometheus has many samples rejected

Got this alert today from Prometheus and had no idea what it meant:

Looking at the Prometheus logs, it was full of similar lines like these:

I did have a clue to go off of — someone told me they just deployed out the same service but to another namespace.

This is our normal regular service name: managedkube/svcmon-managedkube-api

This is the newly added service for a demo : managedkube-demo/svcmon-managedkube-api

That seems to make sense if two of the same services were being scraped and added in without Prometheus changing the labels it would cause this. However, when Prometheus scrapes the endpoints, it would usually add in the namespace label into the metric which would make the metric unique.

I started to poke around in the Prometheus web interface to see if something stands out. Going to the Service Discovery section:

Looking at the service for the duplicate service of the new demo service: managedkube-demo/svcmon-managedkube-api

I noticed that the namespace which should have been managedkube-api-demo it was still managedkube-api .

The problem here is that adding this additional Prometheus Service Monitor it was scraping the old and the new service. Now there are two of the Service Monitors since the deploy deploy this out as well. There were indeed duplicate metrics being put into Prometheus.

The next thing to figure out is why?

Looking at the service monitor definition:

The important part is:

This means that it will go through all of the namespace and pick up anything that matched the labels. This was causing the each service to be scraped twice.

The Service Monitor lives in the namespace of the service. The change and the fix was to make it only select from the namespace that it lived in:

This fixed the problem and the alarm cleared up.

Need personalized help?

ManagedKube provides DevOps consulting services that help you leverage the power of Docker/Kubernetes in building highly resilient, secure, and scalable fully automated CI/CD workflows.

Источник

Error on ingesting samples with different value and the prometheus log does not show #2858

Just like the issue on this page :
#2493

But I did not see the log that indicate the error metric?
I’ve noticed the log in prometheus should be like this?

The text was updated successfully, but these errors were encountered:

This is the part of prometheus logs
time=»2017-06-16T08:47:51Z» level=info msg=»Indexing metrics in memory.» source=»crashrecovery.go:502″ time=»2017-06-16T08:47:51Z» level=info msg=»10000 metrics queued for indexing.» source=»crashrecovery.go:507″ time=»2017-06-16T08:47:51Z» level=info msg=»20000 metrics queued for indexing.» source=»crashrecovery.go:507″ time=»2017-06-16T08:47:51Z» level=info msg=»30000 metrics queued for indexing.» source=»crashrecovery.go:507″ time=»2017-06-16T08:47:51Z» level=info msg=»Indexing archived metrics.» source=»crashrecovery.go:510″ time=»2017-06-16T08:47:51Z» level=info msg=»40000 metrics queued for indexing.» source=»crashrecovery.go:523″ time=»2017-06-16T08:47:51Z» level=info msg=»50000 metrics queued for indexing.» source=»crashrecovery.go:523″ time=»2017-06-16T08:47:51Z» level=info msg=»60000 metrics queued for indexing.» source=»crashrecovery.go:523″ time=»2017-06-16T08:47:51Z» level=info msg=»70000 metrics queued for indexing.» source=»crashrecovery.go:523″ time=»2017-06-16T08:47:52Z» level=info msg=»80000 metrics queued for indexing.» source=»crashrecovery.go:523″ time=»2017-06-16T08:47:52Z» level=info msg=»90000 metrics queued for indexing.» source=»crashrecovery.go:523″ time=»2017-06-16T08:47:52Z» level=info msg=»100000 metrics queued for indexing.» source=»crashrecovery.go:523″ time=»2017-06-16T08:47:52Z» level=info msg=»110000 metrics queued for indexing.» source=»crashrecovery.go:523″ time=»2017-06-16T08:47:52Z» level=info msg=»120000 metrics queued for indexing.» source=»crashrecovery.go:523″ time=»2017-06-16T08:47:52Z» level=info msg=»All requests for rebuilding the label indexes queued. (Actual processing may lag behind.)» source=»crashrecovery.go:529″ time=»2017-06-16T08:47:52Z» level=info msg=»Checkpointing fingerprint mappings. » source=»persistence.go:1480″ time=»2017-06-16T08:47:52Z» level=info msg=»Done checkpointing fingerprint mappings in 1.537957ms.» source=»persistence.go:1503″ time=»2017-06-16T08:47:52Z» level=warning msg=»Crash recovery complete.» source=»crashrecovery.go:152″ time=»2017-06-16T08:47:52Z» level=info msg=»30673 series loaded.» source=»storage.go:378″ time=»2017-06-16T08:47:52Z» level=info msg=»Starting target manager. » source=»targetmanager.go:61″ time=»2017-06-16T08:47:52Z» level=info msg=»Listening on :9090″ source=»web.go:259″ time=»2017-06-16T08:47:53Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=8 source=»scrape.go:520″ time=»2017-06-16T08:47:53Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=7 source=»scrape.go:520″ time=»2017-06-16T08:47:55Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=6 source=»scrape.go:520″ time=»2017-06-16T08:48:05Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=7 source=»scrape.go:520″ time=»2017-06-16T08:48:06Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=8 source=»scrape.go:520″ time=»2017-06-16T08:48:07Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=8 source=»scrape.go:520″ time=»2017-06-16T08:48:08Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=8 source=»scrape.go:520″ time=»2017-06-16T08:48:08Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=7 source=»scrape.go:520″ time=»2017-06-16T08:48:10Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=6 source=»scrape.go:520″ time=»2017-06-16T08:48:20Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=7 source=»scrape.go:520″ time=»2017-06-16T08:48:21Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=8 source=»scrape.go:520″ time=»2017-06-16T08:48:22Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=8 source=»scrape.go:520″ time=»2017-06-16T08:48:23Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=8 source=»scrape.go:520″ time=»2017-06-16T08:48:23Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=7 source=»scrape.go:520″ time=»2017-06-16T08:48:25Z» level=warning msg=»Error on ingesting samples with different value but same timestamp» numDropped=6

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

Источник

December 15, 2022 by Julius Volz

Have you ever encountered any of the following errors in your Prometheus server logs?

«Error on ingesting out-of-order samples»
«Error on ingesting samples with different value but same timestamp»
«duplicate sample for timestamp»

Then you likely have a misconfiguration in your setup that causes multiple series to collide and conflict with each other. In this post, we’re going to explain the background behind these errors, what may cause them, and how to you can debug and fix them.

Background: The TSDB is (normally) append-only

Prometheus is a live monitoring system that usually only needs to keep track of metrics in a steadily ongoing way over time. The Prometheus TSDB reflects this by (normally) only supporting appends to the end of any existing series. That means that it rejects incoming samples that have an earlier timestamp than the latest one within the same series. It also rejects samples with the same timestamp as the latest one in their series when they have a different sample value (otherwise, they are ignored).

However, it is possible to misconfigure Prometheus in such a way that you get duplicated or out-of-order sample timestamps that Prometheus tries and fails to append to the TSDB. This is usually due to an accidental label set collision between multiple time series that causes the series to look the same (same metric name and label set), but it can have other reasons as well. When that happens, appends to the TSDB for the offending samples will fail, and Prometheus will log corresponding errors.

Side note: Prometheus 2.39 introduced an experimental out_of_order_time_window configuration setting that allows ingesting out-of-order samples within a limited time period, but this feature is not enabled by default. It can be useful in certain situations where not all data can be gathered live.

Underlying causes of sample timestamp errors

Let’s have a look at the most common causes for these problems, along with the error messages they produce.

In a correctly configured Prometheus setup, every target has its own unique set of target labels. Prometheus attaches these labels to all series pulled from a target to help disambiguate otherwise identical metrics from different targets. But it’s possible to misconfigure Prometheus in such a way that multiple targets share the same label set, which can cause label set collisions between the resulting time series. The TSDB will then see the streams of multiple original series as a single series, but reject invalid appends when their samples conflict with each other with out-of-order or duplicate timestamps.

Here’s one way to easily provoke this duplication, by explicitly setting the job label of a target group to the same value as another job with an otherwise identical target in your prometheus.yml :

Another variant of the above could be achieved by adding target relabeling rules to the second scrape configuration and overriding the job label to «demo» that way.

When one of these duplicated targets starts scraping before the other one, but finishes the scrape last, you will get an «Error on ingesting out-of-order samples» error message, as the first target tries to insert its samples into the TSDB with a scrape timestamp before the second one that has already completed. As a full log line, that will look something like this:

When two duplicated targets start scraping at exactly the same millisecond (which can and does happen between targets from different scrape pools), they will try storing their resulting samples with the same scrape timestamp. But one of these appends will fail with an «Error on ingesting samples with different value but same timestamp» error message:

Usually the scrape_pool and target fields from the error message should be enough to point you at the right place in the configuration file to find and fix the conflicting targets. However, the duplication might not be visible in the configuration file itself, in case a remote service discovery mechanism produces duplicate targets, or there is an interaction between the service discovery source and target relabeling rules that leads to a loss of disambiguating labels. In that case, you may also want to take a look at the /targets page of your Prometheus server to spot identically labeled targets between multiple scrape pools or scrape configs (it’s not possible to have identical targets in the same scrape pool, as Prometheus automatically deduplicates them into a single one). The /service-discovery page also shows all discovered targets with their label sets before and after relabeling. To fix the ingestion errors, make sure that all targets are labeled uniquely.

Buggy client-side timestamps

Usually Prometheus /metrics endpoints will not include explicit timestamps for the samples they expose. Instead, Prometheus assigns its own scrape start timestamp to all pulled samples. However, it’s also possible to provide explicit client-side timestamps for samples in the Prometheus metrics exposition format. For example, if you wanted to expose a test_metric metric with a value of 42 and a milliseconds Unix timestamp of 1671021749332 , the exposition format would read:

Given the TSDB restrictions mentioned initially, there is a lot that can go wrong here if that feature is misused or incorrectly implemented: if a client-side timestamp moves backwards between scrapes, you will get an out-of-order error. If it stays the same, but the sample value changes, you will get a duplicate timestamp error.

The error messages in this case will look similar to the ones for duplicate targets, as they both originate in the scrape layer. However, the simple target information from the error logs may not be enough anymore to find the specific problematic metric within the broken target. Luckily, setting the —log.level=debug flag on the Prometheus server increases logging verbosity such that you can see each series (in the series log message field) that is causing trouble.

For duplicate timestamps, that will look like this:

And for out-of-order timestamps, like this:

In this case, you will have to fix the metrics exposition in the broken target to resolve the problem.

Duplicate series from rules

It is also possible to create colliding series between the outputs of multiple recording rules. Take this rule file example, which records the metric colliding_name from two rules in the same rule group, but with different sample values:

Although the evaluation of multiple rules from the same group happens sequentially, Prometheus will run them all with the same query evaluation timestamp. This causes the rule evaluation to fail with a «duplicate sample for timestamp» error:

This error also helpfully logs the rule group, the recorded metric name, as well as the offending label set of the sample. That way, you should be able to fix the problem in your rule files.

As a variation, it is also possible for such a collision to happen between series produced from a recording rule and a target scrape.

Remote write sending bad data

If you are using the Prometheus server’s own remote write receiver to push samples to Prometheus, you can also run into situations where one or multiple push sources can send conflicting metrics and timestamps. In that case, you may get an «Out of order sample from remote write» or «duplicate sample for timestamp» request error on the sending side:

And in the receiving Prometheus server, you will get a similar error message (once a fix for a bug that we discovered in the course of writing this article has been fixed):

This latter message tells you the exact offending series, so you can hopefully track down and fix its source.

Intentional ingestion of out-of-order data

Now, there are a few valid use cases when you might actually want to ingest out-of-order data or even data that is normally too old to be ingested into the TSDB at all. You may have some data sources that cannot produce metrics on an ongoing live basis, but only deliver batched or otherwise delayed results. For these cases, as mentioned above, you will want to turn on Prometheus’ new experimental out_of_order_time_window TSDB configuration setting that allows ingesting out-of-order samples within a limited time period to get rid of these error messages.

Caution: Not all collisions cause errors!

As we just learned above, there are multiple series collision conditions that Prometheus can notice and tell us about. But in some cases, Prometheus silently ignores series collisions or can’t detect them at all. So not seeing an error is not a guarantee that everything is working right.

Silent failure of collisions caused by metric relabeling

One such case is when accidentally using metric relabeling (which is applied to each pulled series during a scrape) to remove important disambiguating labels of series from a single target. For example, the following configuration removes the quantile label from all pulled metrics, but in the go_gc_duration_seconds metric, this label is required to tell 3 different series apart:

The current behavior of Prometheus is to store the first of the conflicting samples, but silently discard the subsequent ones without any warning. We filed an issue about possibly reviewing and/or fixing that behavior.

Two rules or targets writing into the same series without timing conflicts

It can easily happen that Prometheus either scrapes two duplicate targets or ingests samples from two rules writing to the same series, but the samples always arrive in perfect lockstep without creating an out-of-order condition or a duplicate timestamp. In that case, the samples from the two sources will be written to the same time series, with the same value altering back and forth between both input series.

For example, if you created a rule file like this, recording the metric colliding_name both with a sample value of 0 and 1 :

. then you will likely not see any logged errors (unless the two rule groups happen to run at the exact same duplicate timestamp). However, you will see the sample value of the series alternating between 0 and 1 :

There’s usually no valid use case for this kind of behavior, but it’s currently not possible for Prometheus to detect this case automatically.

Detecting these issues using metrics

Besides reading logged error messages, you can also detect out of order and duplicate timestamps by monitoring the following metrics that Prometheus exposes about itself:

prometheus_tsdb_out_of_order_samples_total : The total number of out of order samples ingestion failed attempts due to out-of-order support being disabled.
prometheus_target_scrapes_sample_duplicate_timestamp_total : The total number scraped samples with duplicate timestamps.
prometheus_target_scrapes_sample_out_of_order_total : The total number of scraped out-of-order samples.
prometheus_http_requests_total : The total number of 400 status code requests received by the remote write receiver (in the receiving Prometheus server). This can include errors that result in a 400 as well.
prometheus_remote_storage_samples_failed_total : The total number of samples that failed to be sent over remote write in the sending Prometheus server. This can also include other errors.

Series collision errors from rule evaluations unfortunately do not show up in a metric yet. There is a prometheus_rule_evaluation_failures_total metric, but that only counts total failures of a rule’s evaluation, not individual series that have been discarded during the evaluation. However, a rule with problematic output series will still be marked as unhealthy, and the /rules page will show you the corresponding error:

Other storage errors

Besides out-of-order and duplicate sample timestamps, the Prometheus TSDB can also respond with a set of other errors when trying to append samples or even exemplar data. You can see all possible errors in the Prometheus storage code.

Hopefully these explanations were useful for understanding the background behind these out-of-order and duplicate sample ingestion errors and can be helpful in finding and fixing them the next time they occur in your Prometheus setup.

If you would like to learn Prometheus in a structured way and from the ground up, please also be sure to check out our self-paced training courses that cover the basics of Prometheus, as well as advanced use cases and integrations.

Источник

Got this alert today from Prometheus and had no idea what it meant:

Looking at the Prometheus logs, it was full of similar lines like these:

level=warn ts=2019-08-20T20:24:00.084840306Z caller=scrape.go:1091 component="scrape manager" scrape_pool=managedkube/svcmon-managedkube-api/0 target=http://100.96.6.40:5001/ msg="Error on ingesting samples with different value but same timestamp" num_dropped=2
level=warn ts=2019-08-20T20:24:01.302097427Z caller=scrape.go:1091 component="scrape manager" scrape_pool=managedkube-demo/svcmon-managedkube-api/0 target=http://100.96.1.126:5001/ msg="Error on ingesting samples with different value but same timestamp" num_dropped=5

I did have a clue to go off of — someone told me they just deployed out the same service but to another namespace.

This is our normal regular service name: managedkube/svcmon-managedkube-api

This is the newly added service for a demo: managedkube-demo/svcmon-managedkube-api

That seems to make sense if two of the same services were being scraped and added in
without Prometheus changing the labels it would cause this. However, when Prometheus
scrapes the endpoints, it would usually add in the namespace label into the metric
which would make the metric unique.

I started to poke around in the Prometheus web interface to see if something
stands out. Going to the Service Discovery section:

Looking at the service for the duplicate service of the new demo service: managedkube-demo/svcmon-managedkube-api

I noticed that the namespace which should have been managedkube-api-demo it was
still managedkube-api.

The problem here is that adding this additional Prometheus Service Monitor it was
scraping the old and the new service. Now there are two of the Service Monitors since
the deploy deploy this out as well. There were indeed duplicate metrics being put
into Prometheus.

The next thing to figure out is why?

Looking at the service monitor definition:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: svcmon-managedkube-api
    release: prometheus-operator
  name: svcmon-managedkube-api
  namespace: managedkube-api
spec:
  endpoints:
  - interval: 30s
    path: /
    port: prometheus
    scheme: http
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      app: managedkube-api

The important part is:

namespaceSelector:
  any: true

This means that it will go through all of the namespace and pick up anything
that matched the labels. This was causing the each service to be scraped twice.

The Service Monitor lives in the namespace of the service. The change and the fix
was to make it only select from the namespace that it lived in:

namespaceSelector:
  matchName:
  - managedkube-api

This fixed the problem and the alarm cleared up.

Need personalized help?

ManagedKube provides DevOps consulting services that help you leverage the power of Docker/Kubernetes in building highly resilient, secure, and scalable fully automated CI/CD workflows.

Schedule a free 15 minute consultation today by e-mailing us: consulting@managedkube.com

Contact me if you have any questions about this or want to chat, happy to start a dialog or help out: blogs@managedkube.com
{::nomarkdown}

{:/nomarkdown}

kubernetes | prometheus

Источник

I am getting following error in Prometheus on monitoring my Node JS based web application.

level=warn ts=2019-08-02T18:23:48.658364708Z caller=scrape.go:932
component=»scrape manager» scrape_pool=batch_web
target=https://example.com:443/metrics msg=»Error on ingesting samples
that are too old or are too far into the future» num_dropped=6

Any pointers on why this could be happening and how can I solve this?

asked Aug 2, 2019 at 18:27

Enable debugging mode in Prometheus by passing --log.level=debug as an argument to prometheus
Check the log. In my case the metric that was dropped, was returned:
```
msg="Out of order sample" series="some-metric{a="b",c="d"}
```
Check the prometheus configuration and rule files that could return a duplicated some-metric. In my case a duplicated rule was causing the issue as the same metric was scraped from the federated prometheus and calculated again by my prometheus. Removing the duplicated rule solved the issue.

answered Mar 29, 2021 at 12:42

030030

10.3k12 gold badges73 silver badges121 bronze badges

Working with prometheus on virtual machines in VMware, vagrant and Paralles virtual environments, all running Ubuntu 18.04, I had the same promblem with this message in syslog

msg="Error on ingesting out-of-order samples"
and veraious num_dropped= values

The thing that worked was to issue ALL of these command, on each image, then reboot the image. Without the reboot the num_dropped began to be lower but the error persisted until the reboot

/usr/bin/timedatectl set-ntp false
systemctl stop systemd-timesyncd
systemctl disable systemd-timesyncd
systemctl mask systemd-timesyncd

answered Feb 21, 2020 at 21:35

jpietrasjpietras

11 silver badge1 bronze badge

I had the similar issue ( on CentOS7 OS). In my case the /etc/ntp.conf was pointing to a non-existing server, resulting in out-of-sync server time.

In the prometheus server logs I was getting below error message-

target=http://localhost:9090/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=380

Solution:

You can use the below public servers in your /etc/ntp.conf file:

server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

Follow then below steps:

Restart ntp service

systemctl ntpd restart
Stop Prometheus service

systemctl stop prometheus

Go to the Prometheus data directory and remove everything inside it:

ls -lrt
drwxr-xr-x 2 prometheus prometheus     6 Oct 20  2021 chunks_head
-rw-r--r-- 1 prometheus prometheus     0 Oct 19 21:18 lock
-rw-r--r-- 1 prometheus prometheus 20001 Oct 19 21:20 queries.active
drwxr-xr-x 2 prometheus prometheus    66 Oct 19 21:18 wal

Start Prometheus service

answered Oct 24, 2021 at 6:36

Источник

bruno bourdolle

unread,

Nov 16, 2020, 5:35:33 PM11/16/20

to Prometheus Users

hello,

I don(t understand how to correct the WARN about error on ingesting samples…

I’m into kubernetes

I upgrade at last version of prometheus

I cleared the datas

Each time I start I have this line of warning each second

Any idea ?

bets

bruno

level=info ts=2020-11-16T14:29:36.045Z caller=main.go:353 msg=»Starting Prometheus» version=»(version=2.22.1, branch=HEAD, revision=00f16d1ac3a4c94561e5133b821d8e4d9ef78ec2)»

level=info ts=2020-11-16T14:29:36.046Z caller=main.go:358 build_context=»(go=go1.15.3, user=root@516b109b1732, date=20201105-14:02:25)»

level=info ts=2020-11-16T14:29:36.046Z caller=main.go:359 host_details=»(Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 15 17:36:42 UTC 2018 x86_64 prometheus-0 (none))»

level=info ts=2020-11-16T14:29:36.046Z caller=main.go:360 fd_limits=»(soft=1048576, hard=1048576)»

level=info ts=2020-11-16T14:29:36.046Z caller=main.go:361 vm_limits=»(soft=unlimited, hard=unlimited)»

level=info ts=2020-11-16T14:29:36.050Z caller=web.go:516 component=web msg=»Start listening for connections» address=0.0.0.0:9090

level=info ts=2020-11-16T14:29:36.050Z caller=main.go:712 msg=»Starting TSDB …»

level=info ts=2020-11-16T14:29:36.058Z caller=head.go:642 component=tsdb msg=»Replaying on-disk memory mappable chunks if any»

level=info ts=2020-11-16T14:29:36.093Z caller=head.go:656 component=tsdb msg=»On-disk memory mappable chunks replay completed» duration=35.773378ms

level=info ts=2020-11-16T14:29:36.093Z caller=head.go:662 component=tsdb msg=»Replaying WAL, this may take a while»

level=info ts=2020-11-16T14:29:37.850Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=0 maxSegment=17

level=info ts=2020-11-16T14:29:38.244Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=1 maxSegment=17

level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=2 maxSegment=17

level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=3 maxSegment=17

level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=4 maxSegment=17

level=info ts=2020-11-16T14:29:39.839Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=5 maxSegment=17

level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=6 maxSegment=17

level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=7 maxSegment=17

level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=8 maxSegment=17

level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=9 maxSegment=17

level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=10 maxSegment=17

level=info ts=2020-11-16T14:29:39.840Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=11 maxSegment=17

level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=12 maxSegment=17

level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=13 maxSegment=17

level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=14 maxSegment=17

level=info ts=2020-11-16T14:29:39.841Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=15 maxSegment=17

level=info ts=2020-11-16T14:29:39.842Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=16 maxSegment=17

level=info ts=2020-11-16T14:29:39.842Z caller=head.go:714 component=tsdb msg=»WAL segment loaded» segment=17 maxSegment=17

level=info ts=2020-11-16T14:29:39.842Z caller=head.go:719 component=tsdb msg=»WAL replay completed» checkpoint_replay_duration=97.404µs wal_replay_duration=3.748284098s total_replay_duration=3.78422202s

level=info ts=2020-11-16T14:29:39.944Z caller=main.go:732 fs_type=XFS_SUPER_MAGIC

level=info ts=2020-11-16T14:29:39.944Z caller=main.go:735 msg=»TSDB started»

level=info ts=2020-11-16T14:29:39.944Z caller=main.go:861 msg=»Loading configuration file» filename=/etc/prometheus/prometheus.yml

level=info ts=2020-11-16T14:29:39.945Z caller=kubernetes.go:263 component=»discovery manager scrape» discovery=kubernetes msg=»Using pod service account via in-cluster config»

level=info ts=2020-11-16T14:29:39.947Z caller=kubernetes.go:263 component=»discovery manager scrape» discovery=kubernetes msg=»Using pod service account via in-cluster config»

level=info ts=2020-11-16T14:29:40.030Z caller=main.go:892 msg=»Completed loading of configuration file» filename=/etc/prometheus/prometheus.yml totalDuration=85.979364ms remote_storage=5.031µs web_handler=653ns query_engine=1.227µs scrape=250.298µs scrape_sd=2.56867ms notify=33.201µs notify_sd=11.894µs rules=82.217333ms

level=info ts=2020-11-16T14:29:40.030Z caller=main.go:684 msg=»Server is ready to receive web requests.»

level=warn ts=2020-11-16T14:30:00.551Z caller=scrape.go:1372 component=»scrape manager» scrape_pool=kermit-proxy target=https://userprom.xxx:443/metrics msg=»Error on ingesting samples with different value but same timestamp» num_dropped=30

level=warn ts=2020-11-16T14:30:11.443Z caller=scrape.go:1372 component=»scrape manager» scrape_pool=kermit-proxy target=https://userprom.xxx:443/metrics msg=»Error on ingesting samples with different value but same timestamp» num_dropped=30

Matthias Rampke

unread,

Nov 16, 2020, 6:48:25 PM11/16/20

to bruno bourdolle, Prometheus Users

This can happen in a few ways:

1. whatever exports the metrics, does so with a timestamp, but actually changes the value on you between scrapes without updating the timestamp. This is relatively unlikely unless this is something very specialized.

1.1 or it actually exposes the same metric twice within one /metrics response – but the official client libraries actively prevent that

2. (more likely) after all relabeling, you end up with metrics from multiple targets without any distinguishing labels between them.

Are you scraping through some kind of proxy? In general, Prometheus expects to discover and access each target individually, so that it can separately collect the data from all of them.

Look at the raw metrics endpoints that Prometheus would scrape – if they do not have duplicates or timestamps, it must be 2. Remove label drop or replacement rules that may accidentally coalesce multiple targets into having the same labels, especially if you are messing with the «instance» label. Looking at Prometheus’ target page may also help in identifying targets that have the exact same label set.

/MR

bruno bourdolle

unread,

Nov 17, 2020, 5:07:00 PM11/17/20

to Prometheus Users

hi,

I localise the part of the conf that generate the error, but I’m not sure to understand how to correct it. What is strange is that the same conf on dev env works without these warns

group=»production»
instance=»userprom.xxx.intraxxx:443″
job=»my-proxy»

on dev no warns

on prod a lot of warns

if I remove this block, I works without warns on prod , others conf are ok but If I had this one, I have warns on prod

Any idea to correct and explain why, I’m a newbie

Matthias Rampke

unread,

Nov 18, 2020, 12:00:12 PM11/18/20

to bruno bourdolle, Prometheus Users

Is this proxying to more than one backend? And in dev, there is only one?

/MR

bruno bourdolle

unread,

Nov 18, 2020, 12:52:48 PM11/18/20

to Prometheus Users

maybe It ‘s an explication but I’m not the owner of xxx-proxy……How to add a distinct labl/annotation to differentiate them ?

Matthias Rampke

unread,

Nov 18, 2020, 3:13:29 PM11/18/20

to bruno bourdolle, Prometheus Users

Actually, I may have misunderstood – do you want to scrape the proxy itself, or something behind it?

In the former case, and assuming there is no further load balancing involved, you can use a service discovery mechanism that is appropriate in your environment. In a static config you directly provide the target(s). With service discovery, you provide Prometheus with a way to resolve each individual instance and automatically generate the targets. Unfortunately I cannot go into more detail without knowing the specifics of your environment.

/MR

Источник

We are running PMM v1.17.0 and prometheus is causing huge cpu and mem usage (200% CPU and 100% RAM), and pmm went down because of this. We are running PMM on a VM with 2vCPUs and 7.5G RAM, and are monitoring about 25 servers. PMM is running with below command >>

docker run -d -it --volumes-from pmm-data --name pmm-server  -e QUERIES_RETENTION=1095   -p 80:80   -e METRICS_RESOLUTION=3s  --restart always percona/pmm-server:1

The prometheus.log is filled with below entries:

level=warn ts=2020-01-30T10:27:12.8156514Z caller=scrape.go:713 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.21:42002/metrics-mr msg="append failed" err="out of order sample"
level=warn ts=2020-01-30T10:27:26.464361371Z caller=scrape.go:945 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.223:42002/metrics-mr msg="Error on ingesting samples with different value but same timestamp" num_dropped=1
level=warn ts=2020-01-30T10:27:27.81316996Z caller=scrape.go:942 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.21:42002/metrics-mr msg="Error on ingesting out-of-order samples" num_dropped=2
level=warn ts=2020-01-30T10:27:27.813257165Z caller=scrape.go:713 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.21:42002/metrics-mr msg="append failed" err="out of order sample"
level=warn ts=2020-01-30T10:27:41.462420708Z caller=scrape.go:945 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.223:42002/metrics-mr msg="Error on ingesting samples with different value but same timestamp" num_dropped=1
level=warn ts=2020-01-30T10:27:42.813356387Z caller=scrape.go:942 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.21:42002/metrics-mr msg="Error on ingesting out-of-order samples" num_dropped=2
level=warn ts=2020-01-30T10:27:42.813441108Z caller=scrape.go:713 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.21:42002/metrics-mr msg="append failed" err="out of order sample"
level=warn ts=2020-01-30T10:27:56.463798729Z caller=scrape.go:945 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.223:42002/metrics-mr msg="Error on ingesting samples with different value but same timestamp" num_dropped=1
level=warn ts=2020-01-30T10:27:57.82083775Z caller=scrape.go:942 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.21:42002/metrics-mr msg="Error on ingesting out-of-order samples" num_dropped=2
level=warn ts=2020-01-30T10:27:57.820912309Z caller=scrape.go:713 component="scrape manager" scrape_pool=mysql-mr target=https://10.40.4.21:42002/metrics-mr msg="append failed" err="out of order sample"

Can someone please let me know why prometheus is causing issue? Any parameters we need to add/change?

asked Jan 30, 2020 at 10:56

How many server are you monitoring? PMM server of that spec can handle maybe 4-8 monitored servers if they are not too busy. Nearer 4 if they are busy and send a lot of queries to PMM for QAN. It also depends on your data retention, if you increase retention from defaults you will need to add more RAM and CPU to the host.

answered Jun 10, 2021 at 20:54

100% RAM — You are probably swapping, which is terrible for performance. Lower innodb_buffer_pool_size a little to avoid swapping.

200% CPU — Poor indexes and/or poor formulation of queries. Please provide some of the queries and SHOW CREATE TABLE; there may be a quick fix.

«out of order» and «different value» — Either a bug with the collection mechanism or a bug in Percona.

answered Feb 1, 2020 at 19:29

Rick JamesRick James

73.1k4 gold badges41 silver badges101 bronze badges

Источник