Http error bad gateway prometheus grafana

I have Grafana and Prometheus set up on my k8s cluster. Both were installed thru helm using https://github.com/helm/charts/tree/master/stable. Both Grafana and Prometheus are set up thru k8s nginx

I have Grafana and Prometheus set up on my k8s cluster. Both were installed thru helm using https://github.com/helm/charts/tree/master/stable.
Both Grafana and Prometheus are set up thru k8s nginx ingress with my domian addresses.
When I try to set up the Prometheus as a data source in Grafana I get HTTP Error Bad Gateway. In the chrome console in Grafana page I see:

http://grafana.domain.com/api/datasources/proxy/1/api/v1/query?query=1%2B1&time=1554043210.447

Grafana version: Grafana v6.0.0 (commit: 34a9a62)

Grafana data source settings for Prometheus:
URL: https://prometheus.mydomain.com:9090

Access: Server(Default)

Auth:
Basic & TLS Client Auth

What might be wrong and how to debug/fix it?

asked Mar 31, 2019 at 14:47

camel's user avatar

In Grafana data source settings for prometheus database add prometheus service dns and service port. Like below

<prometheus service name>. Namespace. Svc. Cluster. Local:9090

answered Mar 31, 2019 at 17:53

P Ekambaram's user avatar

P EkambaramP Ekambaram

14.3k6 gold badges30 silver badges55 bronze badges

6

if you are run Grafana and Prometheus on docker on your local machine and this will do for the datasource settings

Add host as {host.docker.internal} : {port}

example — http://{host.docker.internal}:9090

answered Jul 17, 2021 at 7:36

Sam's user avatar

SamSam

2,0356 gold badges31 silver badges48 bronze badges

1

When you are trying to add a data source, like Prometheus it is a little bit confusing because they are asking for the http, but you have to put your IP adress.

You just have to open CMD and write ipconfig /all and then look at the IPv4 Direction and then you will have your IP.

So the last part is going to Prometheus and in the URL you have to put:
http://{your_IP}:9090

answered Jan 19 at 12:11

darydaro's user avatar

Содержание

  1. [Bug] Constant 502 Bad Gateway from Grafana After 4.0 Upgrade #6790
  2. Comments
  3. Network Error: Bad Gateway(502) #72
  4. Comments
  5. Footer
  6. Bad gate way in Grafana when connecting to InfluxDB #289
  7. Comments
  8. HTTP Error Bad Gateway when using prometheus #14629
  9. Comments
  10. What Grafana version are you using?
  11. What datasource are you using?
  12. What OS are you running grafana on?
  13. What did you do?
  14. What was the expected result?
  15. What happened instead?
  16. Footer
  17. Connecting to Grafana — InfluxDB Error: Bad Request #20761
  18. Comments
  19. Footer

[Bug] Constant 502 Bad Gateway from Grafana After 4.0 Upgrade #6790

What Grafana version are you using?

What datasource are you using?

What OS are you running grafana on?

Ubuntu 16.04, behind nginx reverse proxy on same system.

Load a dashboard and leave it open for a few minutes.

Not have every panel error on refresh.

  • What happened instead?
    After I leave any dashboard open for more than a few minutes, all panels get errors. Once I refresh the page, the panels load. Did not have any such problems before 4.0 upgrade.

An image or text representation of your metric query

aliasByNode(summarize(removeBelowValue(highestCurrent(sys.*.statsd.gauge-mygauge.total_something, 10), 1), ‘$Interval’, ‘avg’), 1)

The raw query and response for the network request

[pid 6684] write(139, «HTTP/1.0 502 Bad GatewayrnDate: Fri, 02 Dec 2016 00:28:56 GMTrnContent-Length: 0rnContent-Type: text/plain; charset=utf-8rnrn», 125

Grafana itself is returning 502 (not nginx), and even with logging set to Debug for file , there is no meaningful data or errors reported in /var/log/grafana/grafana.log . I realize there’s a lack of good debugging information here, but without logs, I’m not really sure how to narrow it down.

I’m okay with downgrading, but can’t find any documentation with regard to schema changes and if a downgrade would cause things to break. I can provide more debugging information if pointed in the right direction. Thanks!

The text was updated successfully, but these errors were encountered:

Источник

Network Error: Bad Gateway(502) #72

Iam getting this error I save the datasource:

Influxdb 1.7.10 (with flux enabled with INFLUXDB_HTTP_FLUX_ENABLED=true)
Grafana 6.6.2

The text was updated successfully, but these errors were encountered:

Ah seems influxdb is now using over 10GB of memory since I enabled flux. Bit much. This is causing it to crash.

Ah seems influxdb is now using over 10GB of memory since I enabled flux. Bit much. This is causing it to crash.

Hi Barsonax, did that solved your problem? I found myself in the same situations, how did you checked that influxdb is crashing when trying to connect with grafana?

Hi Barsonax, did that solved your problem?

I don’t see disabling flux as a proper solution.

I found myself in the same situations, how did you checked that influxdb is crashing when trying to connect with grafana?

I could see the it crashing in kubernetes because it was using too much memory. Memory usage was much lower without flux.

I could see the it crashing in kubernetes because it was using too much memory. Memory usage was much lower without flux.

Well, then at least I dont feel alone 🙂 It looks like it does many petitions and crashes as well through the web browser when doing a query through Grafana. But I am not sure if this is due influxdb v2, flux, grafana or the extension itself. Works but slow.

Did you took any other approach rather than disabling it? I was thinking in just using influxdb v1 and forget about flux.

not sure what we can do here. lets see if this is an issue with the 7x plugin

© 2023 GitHub, Inc.

You can’t perform that action at this time.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.

Источник

Bad gate way in Grafana when connecting to InfluxDB #289

Please run `microk8s.inspect
inspection-report-20190122_192827.tar.gz
We appreciate your feedback. Thank you for using microk8s.

I cannot get the influxdb to grafana. It always return 502 bad gateway error

The text was updated successfully, but these errors were encountered:

Can you please describe what you are deploying and how it fails? Can you share the manifests you are applying and/or instructions on how your setup is supposed to look like and how it is failing? I basically asking for a way to reproduce the issue you are seeing.

Hello, everybody. I have the same error. I only eval microk8s.enable prometheus and add the port forward to svc/grafana 3000 .

After you enable prometheus and having the pods running, can you access grafana through its ClusterIP (with something like http://10.152.183.153:3000/login ) ?

Can you help me reproduce this? How do you do the port forwarding? Are you running MicroK8s in a VM? Can you share the tarball from microk8s.inspect ? Thanks

Yes, I can connect to grafana and success login, but when I try to see metrics I got a red triangle in the corner of the graph. In network tab I found requests with 502 error code on url http://localhost:3000/api/datasources/proxy/1/api/v1/query?query=1%20-%20avg(rate(node_cpu_seconds_total%7Bmode%3D%22idle%22%7D%5B1m%5D))&time=1574896347 .

Port forwarding command: k port-forward -n kubectl monitoring svc/grafana 3000 . Then I forwarded the port to my computer using ssh.

Источник

HTTP Error Bad Gateway when using prometheus #14629

Read before posting:
Please include this information:

What Grafana version are you using?

What datasource are you using?

prometheus latest docke image

What OS are you running grafana on?

Linux 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

What did you do?

I create a prometheus datasource in grafana web ui

What was the expected result?

success create datasource

What happened instead?

Error with ‘HTTP Error Bad Gateway’

I run grafana and prometheus in docker with almost default configuration. after both started, i check connection in grafana:

i think it means that prometheus endpoint is ready for grafana backend.

then I creete data source in grafana ui like this:

so I think is a bug, or I miss some configuration?

The text was updated successfully, but these errors were encountered:

sorry for my mistake

@aximo Hi, did you find any solution for this problem?

I also suffer the same issue, let me know if you found a solution, thanks!

it’s networking / connection issue. Since it’s not a problem in Grafana we close it.

I have the same problem and if I curl the url from the Grafana back-end (pod) it works perfectly so it isn’t networking related

I’m new to Grafana & Prometheus my url is the /metrics uri on the service which works fine e.g.

I have done the ingress controller deployment from the helmchart which creates a custom metrics service that exposes 9913 on the ingress controller pod

I had the same problem and found that changing access from Server to Browser fixes it. It appears that the grafana server if being prevented from accessing the address:port while the browser is not blocked.

@torkelo I guess you could argue this is a network issue but you could also consider it a Grafana config issue. I prefer to use the Server option for better security.

I too was facing it. I’ve solved it but the solution is a bit hilarious. The solution is-
In the URL field, you need to explicitly write the URL (in my case it was http://localhost:9090). Earlier I was not writing it and there already was the url as watermark!

@souvikhaldar you are absolutely right. Its a water mark and not actual content. Thanks.

I still face this error after explicitly writing the URL (in my case it was http://localhost:9090), finally I resolved it by using http://prometheus:9090 or http:/172.17.0.2:9090 because localhost is not regonized as server host when I’m running both Prometheus and Grafana as containers and you should get the correct IP of prometheus container for grafana to connect to, below is the command line I used:

@jimmycgz Thank you for the answer. It now works.

Maybe it’s too late to answer but I faced the same problem and I solved it by using: host.docker.internal instead of localhost .

just a note, you typed host.internal.docker but it is actually host.docker.internal

I had the same issue being completely new to grafana, just following the tutorial. Changing from «Server» to «Browser» did the trick.

Had same issue. First time on grafana. Solved issue by using URL with IP address of Prometheus Docker Pod (docker inspect ) like so url: http://172.17.0.2:9090
This worked instantly.

I mapped port 8010 to 9090 when running Prometheus (-p 8010:9090).
It finds itself on the default network just as the Grafana container does.

I’m able to connect to it either with:
http://172.17.0.4:9090 (connects to the container directly)
or
http://172.17.0.1:8010 (connects to the mapped port)
or
http://server-ip:8010 (connects to the mapped port)

This solution worked for me

First check if port 9090 is working

sudo lsof -i:9090

then check the promethues logs

sudo docker-compose logs promethues

it showed me error global_scrape timeout greater than scrape_interval

they told to increase the scrape_timeout , but nothing about the scrape_interval in /root/bbb-monitoring/prometheus.yaml
So, I increased them to the same value

sudo docker-compose up -d

this should return

then I set the Data Source -> Promethues url to
http://localhost:9090

Hope this helps someone.

On running this on local minikube cluster, use the cluster address provided in logs to connect to prometheus server from grafana as the Access is Server(default) and Host clearly states : Your access method is Server, this means the URL needs to be accessible from the grafana backend/server.

I had the same problem and found that changing access from Server to Browser fixes it. It appears that the grafana server if being prevented from accessing the address:port while the browser is not blocked.

@torkelo I guess you could argue this is a network issue but you could also consider it a Grafana config issue. I prefer to use the Server option for better security.

@AndrewWPhillips changing the access to Browser worked for me. Thanks

Had same issue. First time on grafana. Solved issue by using URL with IP address of Prometheus Docker Pod (docker inspect ) like so url: http://172.17.0.2:9090
This worked instantly.

also works on your host IP like this HTTP://your_host_ip:9090, note that it’s your host IP, not «localhost», not «127.0.0.1»

use the cluster ip’s so http://172.20.207.29:9090

this is if grafana and prometeus are on kubernetes pods

exact solution for this and all related connection problems between docker images. Connection to prometheus to local app and from grafana to prometheus.
All of these issues is able to solved by using host.internal.docker instead of localhost or local ip

i think it’s problem http Access : select Browser not Server

I had the same issue being completely new to grafana, just following the tutorial. Changing from «Server» to «Browser» did the trick.

I did that but nothing shows up in dashboarh.

i had the same issue . In my case i think it happened because i was running graphana and prometheus in the same terminal so one would automatoically shutdown when the other will start. So obviouly graphana for example couldn’t connect with prometheus because it was off. When i open two different terminals and launched each application seperately it worked

The tip from @souvikhaldar was spot on. 🤦 . That is just an absurd hurdle for new users of Grafana to have to stumble across!
This github issue is Google’s first hit for «grafana bad gateway prometheus».

Please, @torkelo, if you could just put something other there, like «» or something, or actually let the help text there (http://localhost:9090/) be actual text instead, I guess you would help a lot of first time users like myself. @souvikhaldar’s comment currently has 47 👍 ‘s, and that are only the ones that admit to this being the problem!

Wrt. the docker containers — if you run both Prometheus and Grafana with —network=»host» (and then dropping the -p arguments for publishing the ports), it works with http://localhost:9090/ .

i think it’s problem http Access : select Browser not Server

this is saving my life 😀

I was facing this problem because I was trying to access localhost:9090 while Grafana was running in a docker container in a bridge network. 🙂

© 2023 GitHub, Inc.

You can’t perform that action at this time.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.

Источник

Connecting to Grafana — InfluxDB Error: Bad Request #20761

I am trying to view Influxdb2 data in Grafana. I use a python script to upload data into influxdb. This is confirmed to be working:

Influxdb is running on the IP shown in the image above. When I try to connect Grafana I get the error IInfluxDB Error: Bad Request

Following the steps in the documentation the url I input into grafana is the same one as in the image above.

The rest of the config is shown in the image below. Both passwords are the admin token and the database is my bucket ID.

Has anyone managed to get a minimum example working between Grafana and Influxdb2 or is there an obvious step I’m missing?

  • System info: Linux 5.4.0-65-generic x86_64
  • InfluxDB version: 2.0.4

The text was updated successfully, but these errors were encountered:

@C-monC I’m not familiar with the Grafana integration, but based on similar issues I’ve seen I’d recommend trying the instructions I listed at influxdata/kapacitor#2476 (comment) to expose your bucket via our V1 compatibility APIs. You’d then use:

  • Database: name of the database you set for your DBRP resource
  • Username: name you set for your V1 user
  • Password: password you set for your V1 user

If that works for you, could you confirm here? I’ll make sure the Grafana-InfluxQL docs get updated to match the steps that work.

I’ve tried those steps — not sure which parameters must reference new objects and which should be new ones. For instance.
influx v1 dbrp create —org kubOrg —db testdb —rp rp —bucket-id d025cfae0bd77b14 -t asgsgsdfg
—db and —rp would be initialized by that name? Could not find a place to create them in the gui. The bucket is a bucket I’ve already made.

influx v1 auth create —org kubOrg —username test —password testtest —write-bucket d025cfae0bd77b14

Creating the token does not create a token in the gui but it does return a column which says «name/token»

ID Description Name / Token User Name User ID Permissions

In grafana I then add the datasource. Password testtest

The rest of the configuration — password is again testtest:

What happens if you switch «HTTP Method» to «POST»? We’ve seen strange failures when using GET (#20713)

If you have control over the server deployment, you could also run influxd —log-level debug to (hopefully) get some tracing logs on the server-side. If switching the HTTP Method doesn’t work, could you paste any logs you can find here?

Any update on this? Having same issue.

I have the same issue. Is there any newer Information?

Use Custom HTTP Header an not «Basic auth» Header = «Authorization» Value =»Token » important is the Space between the Word Token and your Token.

Use Custom HTTP Header an not «Basic auth» Header = «Authorization» Value =»Token » important is the Space between the Word Token and your Token.

I can confirm that this also fixed it for me. The token is created or copied from influxdb web console, then added as a custom http header in grafana datasource config as per instructions on the link above.

Use Custom HTTP Header an not «Basic auth» Header = «Authorization» Value =»Token » important is the Space between the Word Token and your Token.
Found on http://wiki.webperfect.ch/index.php?title=InfluxDB_2.x:_Error:_Bad_Request_(Grafana_and_InfluxQL)&oldid=2578

I can confirm that this also fixed it for me. The token is created or copied from influxdb web console, then added as a custom http header in grafana datasource config as per instructions on the link above.

This didn’t work for me. After generating a token and pasting it in Custom HTTP Headers with the right format (space between «Token» and ) I’m still getting a Bad Request error.

Use Custom HTTP Header an not «Basic auth» Header = «Authorization» Value =»Token » important is the Space between the Word Token and your Token.

I was having the exact same issue and this fix also worked for me. I also continued using the GET http method, not sure if that had anything to do with it.

The process for connecting Grafana to InfluxDB 2.x and using InfluxQL is documented here: https://docs.influxdata.com/influxdb/v2.1/tools/grafana/?t=InfluxQL

Issue resolved for me as well

Format will be like : Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Note: You have to write token space and then actual token value

Use Custom HTTP Header an not «Basic auth» Header = «Authorization» Value =»Token » important is the Space between the Word Token and your Token.

You can also roll back to v1.8.4 to fix this problem.
https://stackoverflow.com/a/66732230/8712494

Use Custom HTTP Header an not «Basic auth» Header = «Authorization» Value =»Token » important is the Space between the Word Token and your Token.

You absolute legend. Thank you! That space makes complete sence, just annoying how the value is star’d out so you think it’s just the API key it required! «header» «token api_key» with a space. You sir, are amazing.

Wow, I’m shocked this worked. Great job.

© 2023 GitHub, Inc.

You can’t perform that action at this time.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.

Источник

[Bug] Constant 502 Bad Gateway from Grafana After 4.0 Upgrade #6790

Comments

willneumob commented Dec 2, 2016 •

What Grafana version are you using?

What datasource are you using?

What OS are you running grafana on?

Ubuntu 16.04, behind nginx reverse proxy on same system.

Load a dashboard and leave it open for a few minutes.

Not have every panel error on refresh.

  • What happened instead?
    After I leave any dashboard open for more than a few minutes, all panels get errors. Once I refresh the page, the panels load. Did not have any such problems before 4.0 upgrade.

An image or text representation of your metric query

aliasByNode(summarize(removeBelowValue(highestCurrent(sys.*.statsd.gauge-mygauge.total_something, 10), 1), ‘$Interval’, ‘avg’), 1)

The raw query and response for the network request

[pid 6684] write(139, «HTTP/1.0 502 Bad GatewayrnDate: Fri, 02 Dec 2016 00:28:56 GMTrnContent-Length: 0rnContent-Type: text/plain; charset=utf-8rnrn», 125

Grafana itself is returning 502 (not nginx), and even with logging set to Debug for file , there is no meaningful data or errors reported in /var/log/grafana/grafana.log . I realize there’s a lack of good debugging information here, but without logs, I’m not really sure how to narrow it down.

I’m okay with downgrading, but can’t find any documentation with regard to schema changes and if a downgrade would cause things to break. I can provide more debugging information if pointed in the right direction. Thanks!

The text was updated successfully, but these errors were encountered:

Источник

Grafana always report «Error connecting to datasource: Loki: Bad Gateway. 502» #609

Comments

xlloveyun commented May 22, 2019

Describe the bug

Loki start but Grafana always report Error 502.

I tried to build and run from source like below:
$ go get github.com/grafana/loki
$ cd $GOPATH/src/github.com/grafana/loki # GOPATH is $HOME/go by default.
$ go build ./cmd/loki
$ ./loki -config.file=./cmd/loki/loki-local-config.yaml

And the loki logs like below

Then I tried to add http://localhost:3100 to grafana, then got

Environment:

Mac, debian and centos, Grafana in docker, and loki is build from source. I don’t know what happend, I’ve checked the 3100 port has been opened, and iptables is stopped.

Expected behavior

Grafana return success when click the Save&Test button.

The text was updated successfully, but these errors were encountered:

tmikaeld commented Jun 10, 2019

@xlloveyun Did you find out what the problem was? I have the same issue.

cfstyle commented Jun 14, 2019 •

I have the same issue.

tmikaeld commented Jun 14, 2019

@cfstyle You can connect to the IP of the host, like http://155.16.4.68:3100

cfstyle commented Jun 14, 2019

thanks. now it’s working fine ( grafana can connect the loki and I can explore logs).
But the error «404 page not found» is still there.

tmikaeld commented Jun 14, 2019

@cfstyle Well, the promtail endpoint will only return 404 if you’re not doing a query or posting data to it. For example, posting to http://155.16.4.68:3100/api/prom/push would work fine.

kush2100 commented Apr 27, 2020 •

Error connecting to data source: Loki: Bad Gateway. 502
Grafana and Loki are working on same server. Now i want to configure Loki Datasource on public IP like http://1.1.1.1:3100, it will give error Loki: Bad Gateway. 502.
But when connect with local host it will work like http://localhost:3100 or http://127.0.0.1:3100.
Even when i try with http://localhost:3100 get error 404 page not found
Can you please help me on this.

Gooooodman commented Jul 7, 2020

Did you find out what the problem was? I have the same issue
why the issue is closed.
where is the solution .

cyriltovena commented Jul 11, 2020

Solution is make sure loki is healthy and properly configured in Grafana

Gooooodman commented Jul 15, 2020

a matter of time

hamiltont commented Aug 13, 2020 •

That’s your problem. To your docker container, localhost means «inside the same container». Loki is not running inside the container, hence the 502 error.

If you’ve named the loki container, you can take advantage of Docker’s DNS and use this as your datasource URL http://loki:3100 — docker will map loki (or whatever your container name is) to the proper IP for the container running loki. This works best if you’re using docker-compose

If loki is running not in a container, but on your host, you can use Dockers host.docker.internal domain as so http://host.docker.internal:3100 . This will auto-map to the IP address of the host machine (as of docker 18.03+)

paaragon commented Aug 19, 2020

That’s your problem. To your docker container, localhost means «inside the same container». Loki is not running inside the container, hence the 502 error.

If you’ve named the loki container, you can take advantage of Docker’s DNS and use this as your datasource URL http://loki:3100 — docker will map loki (or whatever your container name is) to the proper IP for the container running loki. This works best if you’re using docker-compose

If loki is running not in a container, but on your host, you can use Dockers host.docker.internal domain as so http://host.docker.internal:3100 . This will auto-map to the IP address of the host machine (as of docker 18.03+)

This works perfect for me

coffeeguy1 commented Mar 11, 2021

For those who are using docker-compose you guys have to write like hamiltont said http://loki:3100

If you want to know more about it see Networking in Compose

Источник

Grafana «bad gateway» / Logcli «EOF», while frontend says «status=200» #3524

Comments

liuzhen commented Mar 23, 2021

Describe the bug
when running a (same) large query of |= «ppppppppppppp»:

  • grafana always returns «bad gateway»;
  • logcli returns «EOF»

while in the frontend log, it says:

To Reproduce
Steps to reproduce the behavior:

Expected behavior
grafana or logcli return correct «empty» result

Environment:

  • Infrastructure: minio (3 baremetal, 24 disks); loki (1 frontend, 6 other nodes, all baremetal)
  • Deployment tool: docker

Screenshots, Promtail config, or terminal output
frontend config:

Previously I thought those queries were to large, and tweaked some configs.
Seldom times it returns, and the result stats’s total_bytes does not vary from that of frontend when «bad_gateway» much.

The text was updated successfully, but these errors were encountered:

sergshner commented Apr 24, 2021

I’m having similar problem — for the same query for a short query interval (1-6 hours) it works perfectly fine but when I’m trying to request it in an interval of 6-24h I see Bad gateway in Grafana and EOF in Grafana logs although in Loki I see HTTP 200 status. Here is how it looks like when I’m trying to run the same query using curl:

stale bot commented Jun 2, 2021

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

Joyjey commented Aug 18, 2021 •

Got the same issue
if a request is less than 30s — it works
level=info ts=2021-08-18T18:41:15.347847144Z caller=metrics.go:92 org_id=fake traceID=4e620fbda53652ba latency=slow query=» |= »c694fb71-0ea0-45c7-87e0-7821d9b1a3e3»» query_type=filter range_type=range length=1h0m0s step=14s duration=29.138038734s status=200 limit=30 returned_lines=0 throughput=275MB total_bytes=8.0GB
but if it’s more:
level=info ts=2021-08-18T18:40:18.165030011Z caller=metrics.go:92 org_id=fake traceID=19063be4b11b9919 latency=slow query=» |= »c694fb71-0ea0-45c7-87e0-7821d9b1a3e3»» query_type=filter range_type=range length=1h0m0s step=14s duration=30.479555288s status=200 limit=30 returned_lines=0 throughput=261MB total_bytes=8.0GB

./logcli-linux-amd64 query ‘ |= «c694fb71-0ea0-45c7-87e0-7821d9b1a3e3″‘ —since=1h http://loki:3100/loki/api/v1/query_range?direction=BACKWARD&end=1629311987680557684&limit=30&query=%7Bapp%3D%22dashboard-api-sdk-api%22%7D+%7C%3D+%22c694fb71-0ea0-45c7-87e0-7821d9b1a3e3%22&start=1629308387680557684 Query failed: Get «http://loki:3100/loki/api/v1/query_range?direction=BACKWARD&end=1629311987680557684&limit=30&query=%7Bapp%3D%22dashboard-api-sdk-api%22%7D+%7C%3D+%22c694fb71-0ea0-45c7-87e0-7821d9b1a3e3%22&start=1629308387680557684»: EOF

arul-jegadish commented Nov 13, 2021

The exact issue still exists

igorcoding commented Nov 23, 2021

Facing the same issue with loki-distributed setup and s3 storage. Queries

Sermalenk commented Nov 23, 2021

Same problem. Reopen please.

igorcoding commented Nov 29, 2021

@liuzhen @sergshner @Joyjey @arul-jegadish have you been able to resolve this problem in any way?

The interesting part is in my case that query should have returned zero rows, because nothing is really found, but still getting 502 or EOF

cyriltovena commented Nov 30, 2021

Sounds like timeout issue, either in Grafana or in the proxy in front of Loki,

igorcoding commented Nov 30, 2021

But as I said earlier — we connected grafana directly to query frontend, which seems to be returning a 200 OK response, and still receiving 502. In grafana we have dataproxy.timeout set to 600s. What could be the problem? I tried to get some insights from query-frontend source code, but nothing came to my mind. When looking at logs — everyhting is fine, but as soon as query-fronetnd replies with 200 response I receive 502. And some time earlier query-frontend was able to successfully return even longer requests in terms of duration. Maybe the key point is that there are no result rows in response somehow?

daveram commented Dec 10, 2021

Getting the same thing here on v2.2.1, we started with getting the bad gateway in grafana but confirmed with logcli that when we make our query window larger (last 4 days) we get back this EOF but a 200 OK status. Doesn’t happen until the query finishes and it sure looks like it’s pulling all the data in the stats

Источник

HTTP Error — Bad Gateway #28

Comments

collegeimprovements commented Dec 17, 2016

Both Grafana and Prometheus are up. But when I try to add a data source, I get this error.

The text was updated successfully, but these errors were encountered:

collegeimprovements commented Dec 17, 2016

Oh. It was my mistake. I forgot to change localhost with my server’s name. It looks very cool now 🙂

vegasbrianc commented Dec 20, 2016

Hi @collegeimprovements Glad you found a solution. Thanks for the update!

orngeno commented Sep 11, 2017

Hi
I am new to Prometheus and grafana monitoring tool.

I got this error when i tried to create prometheus data source in grafana.
Can you please help me in creating datasource.
thank you

vegasbrianc commented Sep 11, 2017

Hi @orngeno Try copying the settings in this screenshot. You need:

Access: Proxy
URL: http://prometheus:9090
Make sure to mark default as yes as well

Let me know if not open a new issue. Thanks!

orngeno commented Sep 11, 2017

Hi @vegasbrianc and thanks for your suggestion. I tried with the steps which you have provided. This time it displayed this error.

vegasbrianc commented Sep 11, 2017

@orngeno Strange, Can you open the Docker logs to see what they are saying about Prometheus. You can run docker-compose up without the -d which will show the logs in the foreground. Next, try to login and copy and paste the logs in a new GitHub issue. Cheers

orngeno commented Sep 11, 2017

Thank you @vegasbrian.

GalvinGao commented Oct 4, 2019

Just for anyone who might run into this issue: In my Raspberry Pi 4 Model B, when I use http://localhost:9090 as the URL it is showing HTTP Error Bad Gateway, but when I switch to http://127.0.0.1:9090 it works. Not sure why.

oiar commented Oct 4, 2019

@GalvinGao It’s because your machine cannot resolve localhost to 127.0.0.1.
Which means DNS not work

Источник

Failed to Add Data Source: HTTP Error Bad Gateway #295

Comments

coffebar commented Oct 12, 2020


ClickHouse server version 20.9.3
Grafana v7.2.1

The text was updated successfully, but these errors were encountered:

Slach commented Oct 12, 2020 •

please use localhost:8123 instead of localhost:9000
plugin use http/https protocol for connection

coffebar commented Oct 12, 2020

I have the same Error message when using 8123 port

coffebar commented Oct 12, 2020

I’m not using CHProxy ( it marked as optional in readme )

Slach commented Oct 12, 2020 •

do you have grafana and clickhouse on the same stateful server? or it’s a separate docker containers or Linux machines?

could you run the following command on your grafana server and share the results?

coffebar commented Oct 12, 2020

I realized that CH server crashes after start, sorry to bother. Closed

HardRock4Life commented Sep 9, 2021

@Slach
This is what happens when I run this command on a clickhouse docker container:

However, this is what happens when I try to run the same command on a grafana container

It feels like wget does not exist on it. The attempt to install it is also no good.

Slach commented Sep 9, 2021

@Hukuta look like on grafana container you have alpine based Linux distro

try to run apk add —no-cache wget and replace busybox wget version to full wget

HardRock4Life commented Sep 9, 2021 •

Slach commented Sep 10, 2021

HardRock4Life commented Sep 10, 2021 •

This happens on a Grafana container.

And this is on a Clickhouse container, i.e. it works fine there.

Slach commented Sep 10, 2021

@HardRock4Life clickhouse container and grafana container have different loopback interfaces and 127.0.0.1 it’s not the same IP
please use docker-compose or something else and try to link between containers and connect to IP containers

HardRock4Life commented Sep 10, 2021 •

I cloned the repository from https://github.com/lmangani/docker-clickhouse-grafana and replaced the docker-composer.yaml file with the one from the link above. This is the error it outputs now.

Slach commented Sep 10, 2021

@HardRock4Life
look like you don’t understand how exactly works docker-compose
sorry, I have no time to explain it deeply

UserMMT commented Nov 22, 2021

I met the same error i think you should just add an / at the end of your URL then if you’re using access
-Server you should try to check if the clickhouse-server is reachable by the server where grafana is install
-Browser you should try to check if the clickhouse-server is reachable by your browser using the URL you insert
if so everything should be OK

Источник

Понравилась статья? Поделить с друзьями:
  • Http error 612 ssl connect error
  • Http error 601
  • Http error 599 owncloud
  • Http error 557
  • Http error 555