Error code: 502-backend
[edited by admin: check out this «how to deal with 502 Backend errors» page on our wiki]
My django app is getting this error code, I’ve tried to reload it but the problem is not solved. It’s everything ok in the platform?, is something wrong in my app? how i can know it?
The last log in my server.log are these:
Wed Jan 15 22:41:31 2014 — received message 1 from emperor
2014-01-15 22:41:32 …gracefully killing workers…
2014-01-15 22:41:32 Gracefully killing worker 2 (pid: 3116)…
2014-01-15 22:41:32 Gracefully killing worker 1 (pid: 3113)…
2014-01-15 22:41:32 Wed Jan 15 22:41:32 2014 — received message 1 from emperor
2014-01-15 22:41:35 worker 2 buried after 4 seconds
2014-01-15 22:41:36 worker 1 buried after 5 seconds
2014-01-15 22:41:36 uWSGI: GAME OVER (insert coin)
2014-01-15 22:41:36 chdir(): No such file or directory [core/uwsgi.c line 1450]
2014-01-15 22:41:36 VACUUM: unix socket /var/sockets/cespar.pythonanywhere.com/socket removed.
2014-01-15 22:41:41 Starting uWSGI 1.9.20 (64bit) on [Wed Jan 15 21:41:37 2014]
2014-01-15 22:41:41 compiled with version: 4.8.1 on 29 November 2013 16:59:47
2014-01-15 22:41:41 os: Linux-3.11.0-13-generic #20-Ubuntu SMP Wed Oct 23 07:38:26 UTC 2013
2014-01-15 22:41:41 nodename: glenn-liveweb2
2014-01-15 22:41:41 machine: x86_64
2014-01-15 22:41:41 clock source: unix
2014-01-15 22:41:41 detected number of CPU cores: 4
2014-01-15 22:41:41 current working directory: /etc/uwsgi/vassals
2014-01-15 22:41:41 detected binary path: /usr/local/bin/uwsgi
2014-01-15 22:41:41 !!! no internal routing support, rebuild with pcre support !!!
2014-01-15 22:41:41 using Linux cgroup /mnt/cgroups/cpu/user_types/premium with mode 700
2014-01-15 22:41:41 assigned process 5270 to cgroup /mnt/cgroups/cpu/user_types/premium/tasks
2014-01-15 22:41:41 using Linux cgroup /mnt/cgroups/cpuacct/users/cespar with mode 700
2014-01-15 22:41:41 assigned process 5270 to cgroup /mnt/cgroups/cpuacct/users/cespar/tasks
2014-01-15 22:41:41 using Linux cgroup /mnt/cgroups/memory/user_types/premium with mode 700
2014-01-15 22:41:41 assigned process 5270 to cgroup /mnt/cgroups/memory/user_types/premium/tasks
2014-01-15 22:41:41 uWSGI running as root, you can use —uid/—gid/—chroot options
2014-01-15 22:41:41 chroot() to /mnt/chroots/cespar
2014-01-15 22:41:41 setgid() to 60000
2014-01-15 22:41:41 setuid() to 191112
2014-01-15 22:41:41 limiting number of processes to 64…
2014-01-15 22:41:41 your processes number limit is 64
2014-01-15 22:41:41 your memory page size is 4096 bytes
2014-01-15 22:41:41 detected max file descriptor number: 123456
2014-01-15 22:41:41 building mime-types dictionary from file /etc/mime.types…
2014-01-15 22:41:41 536 entry found
2014-01-15 22:41:41 lock engine: pthread robust mutexes
2014-01-15 22:41:41 thunder lock: disabled (you can enable it with —thunder-lock)
2014-01-15 22:41:41 uwsgi socket 0 bound to UNIX address /var/sockets/cespar.pythonanywhere.com/socket fd 7
2014-01-15 22:41:41 Python version: 2.7.5+ (default, Sep 19 2013, 13:52:09) [GCC 4.8.1]
2014-01-15 22:41:41 Python threads support is disabled. You can enable it with —enable-threads
2014-01-15 22:41:41 Python main interpreter initialized at 0x2232cc0
2014-01-15 22:41:41 your server socket listen backlog is limited to 100 connections
2014-01-15 22:41:41 your mercy for graceful operations on workers is 60 seconds
2014-01-15 22:41:41 setting request body buffering size to 65536 bytes
2014-01-15 22:41:41 mapped 501048 bytes (489 KB) for 2 cores
2014-01-15 22:41:41 Operational MODE: preforking
2014-01-15 22:41:41 WSGI app 0 (mountpoint=») ready in 4 seconds on interpreter 0x2232cc0 pid: 5270 (default app)
2014-01-15 22:41:41 uWSGI is running in multiple interpreter mode
2014-01-15 22:41:41 spawned uWSGI master process (pid: 5270)
2014-01-15 22:41:41 spawned uWSGI worker 1 (pid: 5271, cores: 1)
2014-01-15 22:41:41 spawned 2 offload threads for uWSGI worker 1
2014-01-15 22:41:41 spawned uWSGI worker 2 (pid: 5274, cores: 1)
2014-01-15 22:41:41 spawned 2 offload threads for uWSGI worker 2
2014-01-15 22:41:46 announcing my loyalty to the Emperor…
2014-01-15 22:41:46 announcing my loyalty to the Emperor…
2014-01-15 22:51:06 Wed Jan 15 22:51:06 2014 — received message 1 from emperor
2014-01-15 22:51:06 …gracefully killing workers…
2014-01-15 22:51:06 Gracefully killing worker 1 (pid: 5271)…
2014-01-15 22:51:06 Gracefully killing worker 2 (pid: 5274)…
2014-01-15 22:51:07 worker 1 buried after 1 seconds
2014-01-15 22:51:07 worker 2 buried after 1 seconds
2014-01-15 22:51:07 uWSGI: GAME OVER (insert coin)
2014-01-15 22:51:07 chdir(): No such file or directory [core/uwsgi.c line 1450]
2014-01-15 22:51:07 VACUUM: unix socket /var/sockets/cespar.pythonanywhere.com/socket removed.
2014-01-15 22:51:10 Starting uWSGI 1.9.20 (64bit) on [Wed Jan 15 21:51:09 2014]
2014-01-15 22:51:10 compiled with version: 4.8.1 on 29 November 2013 16:59:47
2014-01-15 22:51:10 os: Linux-3.11.0-13-generic #20-Ubuntu SMP Wed Oct 23 07:38:26 UTC 2013
2014-01-15 22:51:10 nodename: glenn-liveweb2
2014-01-15 22:51:10 machine: x86_64
2014-01-15 22:51:10 clock source: unix
2014-01-15 22:51:10 detected number of CPU cores: 4
2014-01-15 22:51:10 current working directory: /etc/uwsgi/vassals
2014-01-15 22:51:10 detected binary path: /usr/local/bin/uwsgi
2014-01-15 22:51:10 !!! no internal routing support, rebuild with pcre support !!!
2014-01-15 22:51:10 using Linux cgroup /mnt/cgroups/cpu/user_types/premium with mode 700
2014-01-15 22:51:10 assigned process 5395 to cgroup /mnt/cgroups/cpu/user_types/premium/tasks
2014-01-15 22:51:10 using Linux cgroup /mnt/cgroups/cpuacct/users/cespar with mode 700
2014-01-15 22:51:10 assigned process 5395 to cgroup /mnt/cgroups/cpuacct/users/cespar/tasks
2014-01-15 22:51:10 using Linux cgroup /mnt/cgroups/memory/user_types/premium with mode 700
2014-01-15 22:51:10 assigned process 5395 to cgroup /mnt/cgroups/memory/user_types/premium/tasks
2014-01-15 22:51:10 uWSGI running as root, you can use —uid/—gid/—chroot options
2014-01-15 22:51:10 chroot() to /mnt/chroots/cespar
2014-01-15 22:51:10 setgid() to 60000
2014-01-15 22:51:10 setuid() to 191112
2014-01-15 22:51:10 limiting number of processes to 64…
2014-01-15 22:51:10 your processes number limit is 64
2014-01-15 22:51:10 your memory page size is 4096 bytes
2014-01-15 22:51:10 detected max file descriptor number: 123456
2014-01-15 22:51:10 building mime-types dictionary from file /etc/mime.types…
2014-01-15 22:51:10 536 entry found
2014-01-15 22:51:10 lock engine: pthread robust mutexes
2014-01-15 22:51:10 thunder lock: disabled (you can enable it with —thunder-lock)
2014-01-15 22:51:10 uwsgi socket 0 bound to UNIX address /var/sockets/cespar.pythonanywhere.com/socket fd 7
2014-01-15 22:51:10 Python version: 2.7.5+ (default, Sep 19 2013, 13:52:09) [GCC 4.8.1]
2014-01-15 22:51:10 Python threads support is disabled. You can enable it with —enable-threads
2014-01-15 22:51:10 Python main interpreter initialized at 0xcf8cc0
2014-01-15 22:51:10 your server socket listen backlog is limited to 100 connections
2014-01-15 22:51:10 your mercy for graceful operations on workers is 60 seconds
2014-01-15 22:51:10 setting request body buffering size to 65536 bytes
2014-01-15 22:51:10 mapped 501048 bytes (489 KB) for 2 cores
2014-01-15 22:51:10 Operational MODE: preforking
2014-01-15 22:51:10 WSGI app 0 (mountpoint=») ready in 1 seconds on interpreter 0xcf8cc0 pid: 5395 (default app)
2014-01-15 22:51:10 uWSGI is running in multiple interpreter mode
2014-01-15 22:51:10 spawned uWSGI master process (pid: 5395)
2014-01-15 22:51:10 spawned uWSGI worker 1 (pid: 5396, cores: 1)
2014-01-15 22:51:10 spawned 2 offload threads for uWSGI worker 1
2014-01-15 22:51:10 spawned uWSGI worker 2 (pid: 5399, cores: 1)
2014-01-15 22:51:10 spawned 2 offload threads for uWSGI worker 2
2014-01-15 22:51:13 announcing my loyalty to the Emperor…
2014-01-15 22:57:04 announcing my loyalty to the Emperor…
2014-01-15 23:03:46 Wed Jan 15 23:03:46 2014 — received message 1 from emperor
2014-01-15 23:03:46 …gracefully killing workers…
2014-01-15 23:03:46 Gracefully killing worker 1 (pid: 5396)…
2014-01-15 23:03:46 Gracefully killing worker 2 (pid: 5399)…
2014-01-15 23:03:47 worker 1 buried after 1 seconds
2014-01-15 23:03:47 worker 2 buried after 1 seconds
2014-01-15 23:03:47 uWSGI: GAME OVER (insert coin)
2014-01-15 23:03:47 chdir(): No such file or directory [core/uwsgi.c line 1450]
2014-01-15 23:03:47 VACUUM: unix socket /var/sockets/cespar.pythonanywhere.com/socket removed.
but I don’t understand anything.
Can anyone help me?
Thanks a lot
deleted-user-121112
|
5
posts
|
Jan. 15, 2014, 11:03 p.m.
|
permalink
Hi there — looks like that’s our fault, one of our web servers had a problem. I’ve rebooted it and it should be OK now. We’ll investigate the underlying cause when we’re back in the office tomorrow — something similar happened to a different web server earlier on today, so there’s clearly something wrong with at least some of them. Thanks for letting us know!
giles
|
11190
posts
|
PythonAnywhere staff
|
Jan. 16, 2014, 12:15 a.m.
|
permalink
Thanks, I have been able to see my app running 10 minutes ago, but now I think that the server is down again.
I hope you can fix it early.
Thanks.
deleted-user-121112
|
5
posts
|
Jan. 16, 2014, 6:48 a.m.
|
permalink
It should be back up now. Keep us posted, everyone, if you see any other issues…
harry
|
2710
posts
|
PythonAnywhere staff
|
Jan. 16, 2014, 11:25 a.m.
|
permalink
We are having the same problem again… There is no way of working in this situation.
deleted-user-121112
|
5
posts
|
Jan. 16, 2014, 12:03 p.m.
|
permalink
Now it’s ok, it’s a bit crazy, don’t you think?
deleted-user-121112
|
5
posts
|
Jan. 16, 2014, 12:08 p.m.
|
permalink
Sometimes, if you hit “reload web app”, and then, while it’s still reloading (you can see the litter spinner thing), you go and hit the website, you’ll see an error. Could that be it? It’s best to wait until the reload is completely finished…
harry
|
2710
posts
|
PythonAnywhere staff
|
Jan. 16, 2014, 12:13 p.m.
|
permalink
Something is wrong again? I’m getting the same error code. I don’t know why but every time I try to work in your hosting I have the same problem. In my opinion it shouldn’t be so common.
deleted-user-121112
|
5
posts
|
Jan. 19, 2014, 11:30 a.m.
|
permalink
hello
i’have the same problem with web2py (i was synchronising my dropbox, because it’s freeze, i have tried to «reload» and now i get the «502» message
regards
h
deleted-user-135865
|
1
post
|
Jan. 19, 2014, 11:57 a.m.
|
permalink
Hm. it seems to be a problem with the same server as last time. And it doesn’t affect all the apps on that server, only 3 or 4 of them, including @cespar’s for a second time.
I’m going to do a fast-reboot. there will be 2-3 minutes of downtime.
harry
|
2710
posts
|
PythonAnywhere staff
|
Jan. 19, 2014, 12:34 p.m.
|
permalink
We think we’ve identified a potential cause of the problem. Looking for ways to confirm / mitigate…
harry
|
2710
posts
|
PythonAnywhere staff
|
Jan. 20, 2014, 1:44 p.m.
|
permalink
Just encountered the same problem, immediately after restarting my django application
deleted-user-151093
|
3
posts
|
March 1, 2014, 9:59 p.m.
|
permalink
me too, looks like the problem has resurfaced
deleted-user-139153
|
23
posts
|
March 1, 2014, 10:23 p.m.
|
permalink
Just reloaded a couple apps on another account I have here and one of them worked and the other one has the same problem /shrug. Hopefully they’re looking into it now, they’re usually quite quick when problems like this pop up.
deleted-user-139153
|
23
posts
|
March 1, 2014, 11:05 p.m.
|
permalink
On the case…
giles
|
11190
posts
|
PythonAnywhere staff
|
March 1, 2014, 11:51 p.m.
|
permalink
OK, it’s a different web server this time. Same problem. Should be back in 5 mins.
giles
|
11190
posts
|
PythonAnywhere staff
|
March 1, 2014, 11:53 p.m.
|
permalink
Back up — thanks 
deleted-user-139153
|
23
posts
|
March 1, 2014, 11:57 p.m.
|
permalink
Right, we should be back now. The first hit on each affected web app might be a bit slow (it’s kicking them off on an as-needed basis) but everything looks OK now from our end. Let me know if you’re still seeing problems.
This is actually quite odd. Like I said in the other thread, the problem is caused when people make a specific kind of mistake in their WSGI files. (Again, that’s not to try to relinquish our responsibility for keeping PythonAnywhere up and running, we need to make it resilient to this kind of mistake.) But that does make it surprising that it’s hit two of our web servers in a 24-hour period. It’s not a mistake that people make frequently.
More investigation needed. I’ll keep an eye on things, and will start watching all of our web servers rather than just the one I knew was having problems. And we’ll also see if there’s any way we can get paged when this specific error happens in the future. It looks like there’s a clear signature in one of the logs, so we should be able to automate that.
giles
|
11190
posts
|
PythonAnywhere staff
|
March 2, 2014, 12:01 a.m.
|
permalink
@studentrentit — thanks for confirming that!
giles
|
11190
posts
|
PythonAnywhere staff
|
March 2, 2014, 12:02 a.m.
|
permalink
Just checked on my other account (studentrentit is one of my accounts) and this one I can’t get up and running. Is there another webserver that’s affected as well? Thanks.
EDIT: studentrentit is back down again as well
deleted-user-102444
|
27
posts
|
March 2, 2014, 12:55 a.m.
|
permalink
This is really weird. All of the web servers look OK right now. I bounced the one that your awwester apps are on anyway, just in case the symptom in the logs that I thought was a sign of the problem was a sufficient but not necessary condition — could you check if either/both are working again?
giles
|
11190
posts
|
PythonAnywhere staff
|
March 2, 2014, 1:16 a.m.
|
permalink
Both seem to be functioning normal now — thank you!
deleted-user-139153
|
23
posts
|
March 2, 2014, 1:22 a.m.
|
permalink
Excellent, thanks for confirming!
giles
|
11190
posts
|
PythonAnywhere staff
|
March 2, 2014, 1:29 a.m.
|
permalink
Mine is up and running again, Excellent response time !
deleted-user-151093
|
3
posts
|
March 2, 2014, 8:07 a.m.
|
permalink
No problem, thanks for confirming
giles
|
11190
posts
|
PythonAnywhere staff
|
March 2, 2014, 6:46 p.m.
|
permalink
Hi, I have the same problem. I’ve changed the value of DEBUG and my app doesn’t work anymore.
I’ve tried to reload my app but with no results.
What can I do?
Thanks in advance!
deleted-user-123295
|
1
post
|
March 18, 2014, 10:46 p.m.
|
permalink
Hi, I’m getting the same problem. I first changed DEBUG = False on my Django app, restarted the app, and everything worked fine. I then tried to push some other code updates, restart the app, and am now getting a 502-backend error. From the other comments, it looks like this is a PythonAnywhere issue or an issue caused by someone else the same server. Can I get some help as well? Thx.
deleted-user-190060
|
2
posts
|
March 19, 2014, 1:18 a.m.
|
permalink
I just got this same error as well. My site was fine before I uploaded a new version at about 2:50 AM GMT, and since then it’s been down
deleted-user-175980
|
3
posts
|
March 19, 2014, 3:14 a.m.
|
permalink
removed: double post
deleted-user-175980
|
3
posts
|
March 19, 2014, 3:14 a.m.
|
permalink
Same here
deleted-user-139153
|
23
posts
|
March 19, 2014, 3:17 a.m.
|
permalink
I’m seeing a Error code: 502-backend. I created a new web app (manual configuration -> Python 2.7) and stuck with the default WSGI ( ie the «Hello World» app). Despite that, I’m seeing the Error code: 502-backend.
I really don’t think it can be me, beacuse I haven’t even hooked up my code to the app yet/
deleted-user-192438
|
1
post
|
March 19, 2014, 3:35 a.m.
|
permalink
I’m also getting a 502-backend error. This error is only occurring on the 1 subdomain that I restarted.
janis
|
35
posts
|
March 19, 2014, 3:49 a.m.
|
permalink
Sorry about that, everyone. One of our web backends had a problem again, it’s fixed now. We’re working with the guys who wrote the misbehaving code to track down a fix, but it’s proving really hard to isolate the problem.
giles
|
11190
posts
|
PythonAnywhere staff
|
March 19, 2014, 11:53 a.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
i am also getting a 502-backend error . can you explain to me what reason is it? i is not work still
deleted-user-194521
|
13
posts
|
March 19, 2014, 12:01 p.m.
|
permalink
@kinfan I’m taking a look
giles
|
11190
posts
|
PythonAnywhere staff
|
March 19, 2014, 12:03 p.m.
|
permalink
Hmm, it looks fine to me — you have a web app with a default PythonAnywhere «Hello world» page on it.
giles
|
11190
posts
|
PythonAnywhere staff
|
March 19, 2014, 12:04 p.m.
|
permalink
Mine is working, but I had to reload the app & then in my browser hit Ctrl+F5 to force refresh. You might be getting the version that your browser has cached.
deleted-user-175980
|
3
posts
|
March 19, 2014, 12:17 p.m.
|
permalink
That’s a good point. Reloading the app shouldn’t be necessary — the fix we did involved reloading every app on the server — but a force-refresh on your browser might be.
giles
|
11190
posts
|
PythonAnywhere staff
|
March 19, 2014, 12:25 p.m.
|
permalink
Force-refresh did the trick for me, thanks!
janis
|
35
posts
|
March 19, 2014, 12:35 p.m.
|
permalink
Thanks for confirming!
giles
|
11190
posts
|
PythonAnywhere staff
|
March 19, 2014, 2:04 p.m.
|
permalink
i also have this debug
deleted-user-194782
|
1
post
|
March 19, 2014, 3:40 p.m.
|
permalink
The error on your site is a different one — it’s reporting a problem with your web application. Check out the error log for details — there’s a link on the «Web» tab.
giles
|
11190
posts
|
PythonAnywhere staff
|
March 19, 2014, 3:54 p.m.
|
permalink
I seem to be having this same issue. I was working on adding a new app and got the 502-backend error, though I waited till it was finished reloading before visiting my page, and I’m now getting that same error on my other two apps which were working fine earlier today and should be unaffected by the new one. I don’t see anything in my error logs. Help?
vyh
|
2
posts
|
April 12, 2014, 8:57 p.m.
|
permalink
Let me take a look — we did have a problem on one server earlier on today, which is fixed, but perhaps it’s spread to others.
giles
|
11190
posts
|
PythonAnywhere staff
|
April 12, 2014, 9:03 p.m.
|
permalink
OK, yes, one of the web servers is having problems. I’m rebooting it.
giles
|
11190
posts
|
PythonAnywhere staff
|
April 12, 2014, 9:07 p.m.
|
permalink
Thanks, that did it.
vyh
|
2
posts
|
April 12, 2014, 9:18 p.m.
|
permalink
A couple of others were also showing problems — all should be fixed now.
giles
|
11190
posts
|
PythonAnywhere staff
|
April 12, 2014, 9:20 p.m.
|
permalink
I’ll keep an eye on all of them for the next few hours. Not sure what’s causing the problem — it’s an error we’ve seen before, but never on so many servers at once. It happening simultaneously on a bunch of them makes our original diagnosis of the underlying problem seem less likely, too. Hmm.
giles
|
11190
posts
|
PythonAnywhere staff
|
April 12, 2014, 9:22 p.m.
|
permalink
Oh, and thanks for confirming that your sites are back!
giles
|
11190
posts
|
PythonAnywhere staff
|
April 12, 2014, 9:22 p.m.
|
permalink
Same problem here
deleted-user-116060
|
5
posts
|
April 28, 2014, 10:09 a.m.
|
permalink
Investigating…
giles
|
11190
posts
|
PythonAnywhere staff
|
April 28, 2014, 11:44 a.m.
|
permalink
OK, it looks like another problem with one of our web servers. I’ve bounced the affected one and all should be OK now.
giles
|
11190
posts
|
PythonAnywhere staff
|
April 28, 2014, 11:54 a.m.
|
permalink
OK…it works..thanks
deleted-user-116060
|
5
posts
|
April 28, 2014, 1:47 p.m.
|
permalink
I get this Error code: 502-backend page here, it’s been there for roughly 2 hours now. No errors on the dashboard logs.
deleted-user-271183
|
2
posts
|
May 20, 2014, midnight
|
permalink
I’m experiencing the same issue as simplequiz—no errors in the logs but «502-backend» appears when I open my site.
deleted-user-249036
|
1
post
|
May 20, 2014, 1:30 a.m.
|
permalink
Hi,
I’m getting the same error with my Flask web app since last night.
Might be domain name related somehow.
My app shows the 502 error if I use my domain name (www.<domainname>.com) but no error occurs if I use http://<username>.pythonanywhere.com/ — even though both apps have the same wsgi file and the same code-base directory.
- B
deleted-user-235244
|
6
posts
|
May 20, 2014, 7:30 a.m.
|
permalink
It seems to be working now. Sometimes you’ll get the 502 for a short period while your web app is reloading.
glenn
|
8701
posts
|
PythonAnywhere staff
|
May 20, 2014, 9:53 a.m.
|
permalink
Thanks glenn, is working now, but 502 was definitely due to something else aside from reloading because I have been reloading trying both my URLs at the same time all morning. maybe there’s something else going on?
deleted-user-235244
|
6
posts
|
May 20, 2014, 11:12 a.m.
|
permalink
After I sent the last post, I discovered from a colleague that we’d actually had one of our web servers misbehaving and that he had fixed it. That was probably what you were seeing.
glenn
|
8701
posts
|
PythonAnywhere staff
|
May 20, 2014, 3:38 p.m.
|
permalink
Well it’s still not working for me. =/
(for simplequiz, I had to create another account to make it work)
deleted-user-274884
|
1
post
|
May 20, 2014, 9:08 p.m.
|
permalink
Sorry, forgot to clear my browser’s cache. Works fine now, thanks.
deleted-user-271183
|
2
posts
|
May 20, 2014, 9:39 p.m.
|
permalink
Great, glad to hear it
giles
|
11190
posts
|
PythonAnywhere staff
|
May 21, 2014, 10:34 a.m.
|
permalink
hi, i got the error like this
Something went wrong 
This website is hosted by PythonAnywhere, an online hosting environment. Something went wrong while trying to load it; please try again later.
Debugging tips
If this is your PythonAnywhere-hosted site, and you just reloaded it, then the problem might simply be that it hasn’t loaded up yet. Try refreshing this page and see if this message disappears.
If you keep getting this message, you should check your site’s server and error logs for any messages — you can view them from the Web tab inside PythonAnywhere.
If there’s nothing in the logs, and you’re sure your site is OK, then it might be a problem on our side. Drop us a line at support@pythonanywhere.com, in the forums, or using the «Send feedback» link on the site, quoting the error code below.
Error code: 502-backend
please help me
deleted-user-4749819
|
5
posts
|
Nov. 1, 2018, 11:01 a.m.
|
permalink
Did you read the help page linked to at the beginning of this topic? What steps have you taken from that help page to resolve the issue, and what were the results of those steps?
conrad
|
4233
posts
|
PythonAnywhere staff
|
Nov. 1, 2018, 4:41 p.m.
|
permalink
hi, i got the error like this Something went wrong This website is hosted by PythonAnywhere, an online hosting environment. Something went wrong while trying to load it; please try again later.
Debugging tips If this is your PythonAnywhere-hosted site, and you just reloaded it, then the problem might simply be that it hasn’t loaded up yet. Try refreshing this page and see if this message disappears.
If you keep getting this message, you should check your site’s server and error logs for any messages — you can view them from the Web tab inside PythonAnywhere.
If there’s nothing in the logs, and you’re sure your site is OK, then it might be a problem on our side. Drop us a line at support@pythonanywhere.com, in the forums, or using the «Send feedback» link on the site, quoting the error code below.
2019-04-21 17:28:36 Starting uWSGI 2.0.17.1 (64bit) on [Sun Apr 21 17:28:35 2019]
2019-04-21 17:28:36 compiled with version: 5.4.0 20160609 on 15 February 2019 14:46:37
2019-04-21 17:28:36 os: Linux-4.4.0-1075-aws #85 SMP Fri Feb 15 13:52:12 UTC 2019
2019-04-21 17:28:36 nodename: glenn-liveweb7
2019-04-21 17:28:36 machine: x86_64
2019-04-21 17:28:36 clock source: unix
2019-04-21 17:28:36 pcre jit disabled
2019-04-21 17:28:36 detected number of CPU cores: 2
2019-04-21 17:28:36 current working directory: (unreachable)/etc/uwsgi/vassals
2019-04-21 17:28:36 detected binary path: /usr/local/bin/uwsgi
2019-04-21 17:28:36 dumping internal routing table
2019-04-21 17:28:36 [rule: 0] subject: path_info regexp: .svgz$ action: addheader:Content-Encoding:gzip
2019-04-21 17:28:36 end of the internal routing table
2019-04-21 17:28:36 chdir() to /home/dextre30/
2019-04-21 17:28:36 limiting number of processes to 120…
2019-04-21 17:28:36 your processes number limit is 120
2019-04-21 17:28:36 your memory page size is 4096 bytes
2019-04-21 17:28:36 detected max file descriptor number: 123456
2019-04-21 17:28:36 building mime-types dictionary from file /etc/mime.types…
2019-04-21 17:28:36 552 entry found
2019-04-21 17:28:36 lock engine: pthread robust mutexes
2019-04-21 17:28:36 thunder lock: disabled (you can enable it with —thunder-lock)
2019-04-21 17:28:36 uwsgi socket 0 bound to UNIX address /var/sockets/dextre30.pythonanywhere.com/socket fd 3
2019-04-21 17:28:36 Python version: 3.6.6 (default, Aug 22 2018, 20:46:46) [GCC 5.4.0 20160609]
2019-04-21 17:28:36 Set PythonHome to /home/dextre30/.virtualenvs/zoosolution-virtualenv
2019-04-21 17:28:36 Python threads support is disabled. You can enable it with —enable-threads
2019-04-21 17:28:36 Python main interpreter initialized at 0xed2220
2019-04-21 17:28:36 your server socket listen backlog is limited to 100 connections
2019-04-21 17:28:36 your mercy for graceful operations on workers is 60 seconds
2019-04-21 17:28:36 setting request body buffering size to 65536 bytes
2019-04-21 17:28:36 mapped 501384 bytes (489 KB) for 2 cores
2019-04-21 17:28:36 Operational MODE: preforking
2019-04-21 17:28:36 initialized 54 metrics
2019-04-21 17:28:36 WSGI app 0 (mountpoint=») ready in 1 seconds on interpreter 0xed2220 pid: 1 (default app)
2019-04-21 17:28:36 uWSGI is running in multiple interpreter mode
2019-04-21 17:28:36 gracefully (RE)spawned uWSGI master process (pid: 1)
2019-04-21 17:28:36 spawned uWSGI worker 1 (pid: 3, cores: 1)
2019-04-21 17:28:36 spawned 2 offload threads for uWSGI worker 1
2019-04-21 17:28:36 spawned uWSGI worker 2 (pid: 6, cores: 1)
2019-04-21 17:28:36 metrics collector thread started
2019-04-21 17:28:36 spawned 2 offload threads for uWSGI worker 2
2019-04-21 17:28:39 DAMN ! worker 2 (pid: 6) died, killed by signal 7 trying respawn …
2019-04-21 17:28:39 Respawned uWSGI worker 2 (new pid: 12)
2019-04-21 17:28:39 spawned 2 offload threads for uWSGI worker 2
2019-04-21 17:28:43 DAMN ! worker 1 (pid: 3) died, killed by signal 7 trying respawn …
2019-04-21 17:28:43 Respawned uWSGI worker 1 (new pid: 16)
2019-04-21 17:28:43 spawned 2 offload threads for uWSGI worker 1
2019-04-21 17:28:44 DAMN ! worker 2 (pid: 12) died, killed by signal 7 trying respawn …
2019-04-21 17:28:44 Respawned uWSGI worker 2 (new pid: 20)
2019-04-21 17:28:44 spawned 2 offload threads for uWSGI worker 2
2019-04-21 17:28:46 DAMN ! worker 1 (pid: 16) died, killed by signal 7 trying respawn …
2019-04-21 17:28:46 Respawned uWSGI worker 1 (new pid: 25)
2019-04-21 17:28:46 spawned 2 offload threads for uWSGI worker 1
2019-04-21 17:28:47 DAMN ! worker 2 (pid: 20) died, killed by signal 7 trying respawn …
2019-04-21 17:28:47 Respawned uWSGI worker 2 (new pid: 28)
2019-04-21 17:28:47 spawned 2 offload threads for uWSGI worker 2
Error code: 502-backend please help me
deleted-user-5146497
|
7
posts
|
April 21, 2019, 5:36 p.m.
|
permalink
That is the server log. Did you also take a look at your error log?
conrad
|
4233
posts
|
PythonAnywhere staff
|
April 21, 2019, 6:03 p.m.
|
permalink
Yes, there is nothing in the error log.
deleted-user-5146497
|
7
posts
|
April 21, 2019, 6:18 p.m.
|
permalink
What does your error log say?
conrad
|
4233
posts
|
PythonAnywhere staff
|
April 21, 2019, 9:25 p.m.
|
permalink
Hey, I got the same error when I tried to enter the admin page. What should i do?
sfmyazilim
|
1
post
|
Feb. 5, 2020, 6:19 a.m.
|
permalink
Are you using Django 3.0 or higher, with Python 3.7? Your account is configured to use Python 3.7.0 as its specific version of 3.7, and that Python version has a bug which makes it crash in admin views if you use any version of Django after 3.0.
There are three ways to fix that:
- We can update your account by changing your system image so that you get 3.7.5 instead of 3.7.0, which will fix the issue. However, because changing the system image upgrades a lot of the pre-installed Python packages, any code that you have that relies on those packages might break if it’s not compatible with the new versions. Also, because the new image has newer versions of Python, if you have any virtualenvs, you may need to rebuild them. If you’re happy to go ahead despite that, just let us know and we’ll switch you over.
- You can switch to using Python 3.6 instead of 3.7
- You can use a version of Django that’s older than 3.0.
The best option is probably to update the system image, so I’d recommend going that way.
giles
|
11190
posts
|
PythonAnywhere staff
|
Feb. 5, 2020, 12:31 p.m.
|
permalink
Ошибка 502 при открытии сайта может появиться неожиданно. В этой статье мы расскажем, что значит код ошибки 502 и что может сделать пользователь и владелец сайта, чтобы её исправить.
Ошибка 502 Bad Gateway: что значит
Файлы любого сайта находятся на физическом сервере. Чтобы их получить и отобразить веб-ресурс на компьютере, браузер делает запрос на сервер. Если он по какой-либо причине не передал файлы, появляется ошибка 500-511.
Ошибка 502 Bad Gateway возникает при неправильной работе прокси-сервера, DNS-сервера и чаще всего сервера, на котором размещён сайт. Проблема может распространяться как на весь ресурс, так и на отдельные страницы. Это зависит от характера проблемы. Существуют разновидности 502 ошибки: Bad Gateway Nginx, Bad Gateway Apache. Об их отличиях мы расскажем ниже. Также эта ошибка может иметь формулировки:
- Bad Gateway: Registered endpoint failed to handle the request, Temporary Error (502),
- Error 502,
- Bad 502 Gateway,
- 502 Error,
- 502. That’s an error,
- 502 Service Temporarily Overloaded,
- 502 Server Error: The server encountered a temporary error and could not complete your request,
- 502 – Web server received an invalid response while acting as a gateway or proxy server,
- 502 Bad Gateway Nginx,
- 502 Proxy Error,
- HTTP 502,
- HTTP Error 502 Bad Gateway.
Что значит плохой шлюз: ошибка 502
Причины возникновения ошибки 502 Bad Gateway
-
Первая и основная причина ― перегрузка сервера. Перегрузка может быть вызвана несколькими проблемами:
- Большое количество посетителей одновременно. Веб-ресурс может посещать ограниченное количество посетителей. Сколько человек может посетить сайт зависит от возможностей сервера (размера оперативной памяти) и настроек, которые сделал создатель ресурса. Если по какой-либо причине на сайт зайдёт больше пользователей, чем запланировано, сервис может не справиться и страница выдаст код 502. Такое случается при рекламных акциях и распродажах в интернет-магазинах.
- Атака хакеров или DDoS-атака. Эта проблема связана с предыдущей причиной перегрузки. Хакер имитирует большой наплыв пользователей, из-за чего сервер выходит из строя. Такие атаки могут быть использованы для снижения продаж.
- Плохая оптимизация сайта. Настройки ресурса сделаны так, что маленькое количество посетителей генерирует много запросов. В этом случае нужно оптимизировать работу сервера с пользовательскими запросами.
- Второй причиной возникновения кода 502 могут явиться ошибки РНР. Если для расширения функционала сайта в панель управления были добавлены некорректно настроенные плагины, они могут выдавать проблемы в своей работе. Вместе с ними ошибку покажет и сайт целиком. Также если код сайта написан неправильно, запросы могут давать отрицательный результат.
- Ошибка браузера. Проблема может быть на стороне пользователя, если у него установлены расширения, которые нарушают соединение с сервером сайта.
Чем отличается ошибка 502 Bad Gateway Nginx
Между браузером и сервером может стоять веб-сервер. Он используется для снижения нагрузки на сервер, аутентификации пользователей и многого другого. Самые популярные программы для создания веб-сервера ― Nginx и Apache. Так как веб-сервер является посредником между браузером и сервером, то именно он будет оповещать пользователя о проблеме. Поэтому в зависимости от веб-сервера в сообщении вы можете увидеть надпись Bad Gateway Nginx или Bad Gateway Apache. При этом причины возникновения проблемы одинаковы.
Как исправить ошибку 502
Что делать, если вы пользователь
- Перезагрузите страницу, если проблема была вызвана наплывом посетителей. Возможно, через некоторое время посетители уйдут со страницы и вы сможете увидеть контент.
- Попробуйте зайти на другой веб-ресурс. Если вы можете зайти на другой сайт, значит проблема на стороне владельца ресурса и вы ничего не можете сделать. Вернитесь на страницу позже, когда администратор восстановит доступ.
- Проверьте подключение к интернету. Из-за низкой скорости или нестабильности соединения браузер может не получать данные с сервера.
- Запустите браузер в режиме «Инкогнито». В режиме «Инкогнито» браузер работает с базовыми настройками. Если вам удалось зайти на веб-ресурс в этом режиме, значит одно из ваших расширений браузера мешает соединению. Это расширение нужно отключить.
- Почистите файлы cookies. Если при повторном входе на сайт всё равно отображается ошибка 502, очистите кэш браузера. Возможно, доступ уже восстановлен, но ваш браузер обращается к старой версии страницы из кэша.
- Очистите кэш DNS. DNS-кэш — это временная база данных вашего компьютера, которая хранит записи обо всех последних посещениях и попытках посещений веб-сайтов и их IP-адресах. Кэш позволяет ускорить вход на часто посещаемые веб-ресурсы. Если у сайта изменились DNS, а данные из кэша отправляют на старый IP-адрес, в браузере появится код 502. После очистки браузер начнёт обращаться к новому IP-адресу.
Как очистить кэш DNS
В зависимости от вашей операционной системы очистите кэш по одной из инструкций.
- Откройте командную строку. Для этого введите в поисковую строку «Командная строка» и выберите появившееся приложение:
- Введите команду:
ipconfig /flushdns
- Дождитесь сообщения об очистке кэша:
- Откройте терминал клавишами Ctrl+Alt+T.
- Введите команду:
Для Ubuntu:
sudo service network-manager restart
Для других дистрибутивов:
sudo /etc/init.d/nscd restart
- Войдите в терминал. Для этого нажмите клавиши Command + Space. Введите Терминал и нажмите на найденное приложение.
- Введите команду:
sudo killall -HUP mDNSResponder
Готово, вы очистили кеш DNS. Попробуйте заново зайти на сайт.
Что делать, если вы владелец сайта
Проверьте количество свободной памяти. Это можно сделать двумя способами.
Способ 1 ― введите команду top в командной строке сервера:
Mem ― вся оперативная память.
Swap ― раздел подкачки.
Посмотрите на строку Mem ― free. Это количество свободного места на сервере. Если там указано маленькое число, ошибка 502 Bad Gateway появляется из-за нехватки памяти. Увеличьте количество оперативной памяти и проблема пропадёт. Также в результатах можно будет увидеть, какую нагрузку на сервер даёт каждый отдельный процесс.
Способ 2 ― введите команду free -m.
Mem ― вся оперативная память.
Swap ― раздел подкачки.
В строке Mem ― free показано свободное место на сервере. Если там маленькое число, увеличьте количество оперативной памяти.
Проверьте логи сервера. Если проблема возникла в момент каких-либо обновлений на сайте, проверьте журнал изменений, чтобы отменить те доработки, которые нарушили функциональность сервера. Также в логах можно увидеть DDos-атаку. Если дело в нехватке памяти, в логах отобразится ошибка OOM (out of memory).
Проверьте плагины в WordPress. Если ваш сайт создан на WordPress, некоторые плагины и темы могут нарушать работу сервера.
-
1.
Войдите в панель управления WordPress. Если вы пользуетесь услугой REG.Site, войти в панель управления CMS можно прямо из Личного кабинета.
-
2.
Перейдите во вкладку «Плагины» ― «Установленные».
-
3.
Нажмите Деактивировать у плагина, который, как вам кажется, повлиял на работу сайта:
Можно сразу отключить все плагины, чтобы убедиться, что один из них влияет на работу сервера. И далее по очереди включайте плагины, пока не найдёте конкретный плагин-виновник.
Проверьте, как работают вспомогательные службы, например MySQL и Memcached. Иногда они могут стать причиной 502 ошибки.
Свяжитесь со службой поддержки своего хостинг-провайдера. Если ничего из вышеперечисленного не помогло, обратитесь к службе поддержки и подробно опишите проблему и действия, которые вы предприняли до обращения. Действуйте по одной из инструкций ниже.
Сайт находится на виртуальном хостинге REG.RU
Если вы столкнулись с единичными случаями возникновения 502 ошибки, можете проигнорировать их.
Если код 502 возникает регулярно, напишите заявку в службу поддержки. В заявке укажите:
- Точное московское время наблюдения проблемы.
- Название сайта, на котором была замечена проблема.
- Если ошибка отображается не сразу, а после определённых действий (добавление изображения, отправка формы с сайта, импорт файлов), подробно опишите порядок действий, по которому мы сможем воспроизвести проблему.
- Если для воспроизведения проблемы необходимо авторизоваться в административной части сайта, предоставьте логин и пароль для доступа.
Сайт находится на VPS REG.RU
Чаще всего на VPS используется связка: Nginx + бэкенд-сервер (Apache, PHP-FPM, Gunicorn, NodeJS). Ошибка 502 возникает в случае, если Nginx не может получить ответ от этих сервисов.
Клиенты с VPS сталкиваются с «502 Bad Gateway», когда:
- какой-то из сервисов выключен. Перезапустите веб-сервер Apache, PHP-FPM либо другой сервис, с которым работает Nginx;
- между Nginx и бэкенд-сервером некорректно настроена связь. Например, Nginx производит обращение к порту 8080, а веб-сервер Apache «слушает» на 8081. В этом случае необходимо скорректировать настройки веб-сервера.
Если вам не удалось самостоятельно устранить ошибку 502, обратитесь в техподдержку. В заявке укажите:
- Точное московское время наблюдения проблемы.
- Название сайта, на котором была замечена проблема.
- Если ошибка отображается не сразу, а после определённых действий (добавление изображения, отправка формы с сайта, импорт файлов), подробно опишите порядок действий, по которому мы сможем воспроизвести проблему.
- Если для воспроизведения проблемы необходимо авторизоваться в административной части сайта, предоставьте логин и пароль для доступа.
You’re viewing Apigee Edge documentation.
View
Apigee X documentation.
Symptom
The client application gets an HTTP status code of 502 with the message
«Bad Gateway» as a response for API calls.
The HTTP status code 502 means that the client is not receiving a valid response from the
backend servers that should actually fulfill the request.
Error Messages
Client application gets the following response code:
HTTP/1.1 502 Bad Gateway
In addition, you may observe the following error messages:
<html>
<head>
<title>Error</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>An error occurred.</h1>
<p>Sorry, the page you are looking for is currently unavailable.<br/>
Please try again later.</p>
</body>
</html>
If the error comes from the backend server, then you may see something like this. The error message from backend completely depends on its implementation.
<html> <head><title>502 Bad Gateway</title></head> <body bgcolor="white"> <center><h1>502 Bad Gateway</h1></center> </body> </html>
Possible Causes
Here are a few possible causes that can lead to 502 Bad Gateway error for APIs going through Apigee Edge:
| Cause | Description | Troubleshooting Instructions Applicable For |
| No MPs available in the pool | This error is observed if all the MPs in the pool are unavailable, that is, they are either down or busy and hence not responding. | Edge Private Cloud users |
| Incorrect SSL configuration between Routers and MPs | This error is observed if the client’s CA signed root certificate is missing in the truststore of Edge’s Router. | Edge Private Cloud users |
| Error from the backend server | This error will be observed if the backend server fails and sends this response. | Edge Public and Private Cloud users |
Cause: No MPs available in the pool
This error will occur if Router finds that all the Message Processors in a given region/data center are unavailable (for example, if they are all down).
Apigee Edge is configured in such a way that the incoming API traffic (requests) in a given region/data center are always routed from the Routers to the Message Processors (MPs) in the same region/data center. In some cases, Apigee Edge components may be setup in just one region/data center and in some cases, they might be setup in more than one region/data center. In each region/data center there will be two or more Routers and Message Processors configured.
Diagnosis
- Determine the region/data center (s) in which the API requests are failing with 502 Bad Gateway error, if there is more than one region/data center. You can find this either by identifying the region in which the users are observing 502 errors or by checking the NGINX Access logs in
/opt/apigee/var/log/edge-router/nginx/directory on each of the Routers belonging to different regions. - You will see the following error in the NGINX Error logs (
/opt/apigee/var/log/edge-router/nginx/ORG-Env._error_log)2019/06/24 15:26:00 [error] 4796#4796: *56357443 no live upstreams while connecting to upstream, client: <Router_IP_address>, server: <HostAlias>, request: "PUT <BasePath> HTTP/1.1", upstream: "http://<ListOfMP-IP_R-MP-Port>/<BasePath>", host: "<HostAlias>"
Scenario 1: All the Message Processors are down
- Check if the Message Processors in the specific region/data center are up and running.
- If all the Message Processors are down, restart them.
Resolution
Restart all the Message Processors using the following command:
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
Scenario 2: All the Message Processors are busy processing ongoing requests
This error will occur if the Routers finds that all the all the Message Processors in a given region/data center are unavailable as they are all busy processing ongoing requests.
- Check if the Message Processors in the specific region/data center are up and running.
- If all the Message Processors are up and active, then check if the Message Processor(s) is experiencing high CPU usage, then generate three thread dumps every 30 seconds using the following command:
<JAVA_HOME>/bin/jstack -l <pid> > <filename>
- If the Message Processor(s) is experiencing high memory usage then generate a heap dump using the following command:
sudo -u apigee /bin/jmap -dump:live,format=b,file=
- Restart the Message Processor using the below command. It should bring down the CPU and Memory:
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
- Monitor the API calls to confirm if the problem still exists.
- Contact Apigee Support and provide the thread dumps, heap dump, and Message Processor logs (
/opt/apigee/var/log/edge-message-processor/logs/system.log)to help investigate the cause for the high CPU/memory usage.
Cause: Incorrect SSL configuration between Routers and MPs
Diagnosis
- Check the NGINX Access logs (
/opt/apigee/var/log/edge-router/nginx/ORG-Env._access_log). You will see 502 response as shown below:2019-07-23T12:13:42+03:00 sc-10-254-226-23 10.X.X.X:53634 10.X.X.X:8998 0.000 - - 502 502 189 344 GET <path> curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 <host alias> mp-10-254-226-23-23706-8552529-1 10.129.107.101 - - -1 - - dc-2 gateway-2 green - gateway-2 dc-2 op pilot http -
- Check the NGINX Error logs (
/opt/apigee/var/log/edge-router/nginx/ORG-Env._error_log). You will see errors like this:2019/07/30 17:02:24 [error] 7691#7691: *11753633 peer closed connection in SSL handshake while SSL handshaking to upstream, client: X.X.X.X, server: <HostAlias>, request: "GET /no-target HTTP/1.1", upstream: "https://X.X.X.X:8998/no-target", host: "<HostAlias>"
- This shows the SSL handshake fails between Router and Message Processor.
- If you notice carefully in the error message in step #1 and #2, the port # used for communicating with the Message Processor is 8998 which is a non secure port but the protocol is SSL (https). Usually the secure port # used is 8443. Since a non secure port is used for secure communication it causes the SSL handshake failure.
- Typically this can happen if you have missed out any steps or set any incorrect values while configuring SSL between Router and Message Processor. Refer to the steps outlined here.
For example, this error can occur if- The port # is specified as 8998 instead of 8443 in
/opt/apigee/customer/application/message-processor.properties as shown belowconf/message-processor-communication.properties+local.http.port=8998
- The Router config files under the directory
/opt/nginx/conf.d/*are not deleted and the Router has not been restarted while doing the SSL configuration. In this scenario, you can notice that the port# of the Message Processors will remain 8998 in the config files.
- The port # is specified as 8998 instead of 8443 in
Resolution
- Ensure that all the steps provided in Configuring TLS between a Router and a Message Processor are followed properly.
- If the problem persists, go to Gather Diagnostic Information.
Cause: Error from the backend server
Diagnosis
- If the error occurs every time, then you can capture the UI trace for the failing requests. Select a failing request and navigate through various phases in the trace. If you notice that you get the “502 Bad Gateway” from the backend server itself, then the issue could be because some failure could have happened on the backend server.
Trace showing 502 Bad Gateway coming from the backend server
- If the issue is intermittent and you are unable to capture the trace,
- If you are a Public Cloud user, then you could use API Monitoring and check the details about the 502 errors.
- If you observe the Fault Code is
messaging.adaptors.http.flow.ErrorResponseCodeand the Fault Source istarget, then the error is caused by the backend server.
- If you observe the Fault Code is
- If you are a Private Cloud user, then you could analyze the NGINX Access logs
/opt/apigee/var/log/edge-router/nginx/ORG-Env._access_log.
You will see the entry for the failing request as follows:2017-02-24T14:42:12+00:00 rt-01 192.8.155.2:18118 192.168.84.166:8998 10.225 - - 502 502 440 0 GET /adv-eadlg-test/documents?type=doctype HTTP/1.1 rt-02efawae234-1234 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 myorg-dev.apigee.net rt-02efawae234-1234 6 - false target messaging.adaptors.http.flow.ErrorResponseCode null/null - /organizations/myorg/environments/dev/apiproxies/api123
- If you observe the Fault Code is
messaging.adaptors.http.flow.ErrorResponseCodeand the Fault Source istarget, then the error is caused by the backend server.
- If you observe the Fault Code is
- If you are a Public Cloud user, then you could use API Monitoring and check the details about the 502 errors.
Resolution
- Work with your backend server team to fix this issue in the backend.
Gather Diagnostic Information
- NGINX Access logs
(/opt/apigee/var/log/edge-router/nginx/ORG-Env._access_log)
and Error logs
(/opt/apigee/var/log/edge-router/nginx/ORG-Env._error_log). - Message Processor logs
(/opt/apigee/var/log/edge-message-processor/logs/system.log).
Maintaining a server is hard.
You have to deal with all the upgrades, security patches and the occassional server errors (aka errors from hell).
One such common error in Nginx servers is “502 Bad Gateway“.
3 word error message – because Nginx doesn’t love you. That’s why.
The error message is cryptic.
So, many web masters roll up their sleeves and look at the error log:
2017/04/04 08:34:43 [error] 949#949: *7 connect() failed (111: Connection refused) while connecting to upstream, client: XXX.XXX.XXX.XXX, server: myserver.com, request: "GET /myurl-this/ HTTP/1.0", subrequest: "/redis-fetch", upstream: "redis://127.0.0.1:6379", host: "refserver.com", referrer: "http://referalsite.com/myurl-this/"
Yeah, more gibberish.
You know something is messed up, because it says “failed” and “refused“.
But WHAT? You hardly have time to get a PhD in computer science.
Here’s help. We’ve listed the top 5 reasons for 502 Bad Gateway error, and how we fix them.
1. Backend service failed
Nginx depends on backend services like PHP-FPM, database services and cache servers to run web applications.
So, if any of these services crash or freeze, Nginx won’t get any data from them, resulting in “502 Bad gateway” error.
Some services that we’ve seen to fail are:
- PHP
- Apache
- Cache
- Database
The reasons for service failure can range from traffic spikes and resource limits to disk errors and DDoS attacks.
If you suspect a backend service is unresponsive or failed, you can try killing all unresponsive processes and restarting the service.
For instance, here’s one way we kill defunct PHP-FPM processes and restart services.
# kill -9 $(pgrep php-fpm)
# /etc/init.d/php-fpm restart
* Restarting PHP FastCGI Process Manager php-fpm [ OK ]
Warning : Do not use these commands if you are not sure how it works.
If the service restart didn’t work, you may need to get someone to take a closer look at the server health.
Our Nginx experts are online 24/7. Click here if you need help resolving your server error.
2. High server load
The second most common reason for “502 bad gateway” in Nginx is high load average in backend servers.
Load spikes cause services to not respond.
We’ve seen these reasons for load spikes:
- Sudden spike in website traffic (can be seasonal or marketing / promotional).
- Malware infection on the server.
- Comment spamming or other vulnerability exploits.
- Brute force attacks that’s designed to exploit web apps.
- Application bugs that cause memory leaks or resource hogging.
To troubleshoot a high load issue, first we figure out which resource is being abused (I/O, Memory, CPU or Net).
The we find out which service is abusing that resource, and from that point, find out which user in that service owns the abusive script or software.
Click here to know more about high load troubleshooting.
If your server is currently under high load, and you need urgent help, click here to contact our Emergency Server Support techs. We are online 24/7 and can help you in a few minutes.
3. Incorrect service configuration
Your Nginx server and the backend services relies on many sub-systems to work properly.
This includes DNS resolution, Apache processes, PHP services, DB server, etc.
If even one of these services have a wrong config entry, that service will fail to respond, and Nginx will show “502 bad gateway” error.
Some configuration issues that we’ve seen are:
- DNS resolver misconfigured in Nginx causing domain lookups to fail.
- DB login details set incorrectly after a recent migration, restore or upgrade.
- Apache firewall settings (mod_security) syntax error causing Apache to crash.
- Incorrect memory or file limits set for PHP applications.
- Capacity limits (like no: of connections per IP) set too restrictively causing legit visits to fail.
- ..and more
There is no easy way to find out a configuration error.
You really need to scan the error log and pay attention to what the error says.
For eg. this error here says the PHP application reached the maximum limit of processes (defined by pm.max_children setting) allowed.
WARNING: [mysite.com] server reached max_children setting (30), consider raising it
ERROR: unable to read what child say: Bad file descriptor (9)
If you are not familiar with PHP or web server settings, it is best to ask a server administrator.
If you need help fixing a similar error, click here to talk to our Nginx admins. We are online 24/7 and can attend your ticket within a few mins.
How Bobcares prevents configuration errors
As a quick aside, here’s how we prevent server errors related to config issues.
Configuration errors are generally caused by stale server settings that’s not adjusted for new traffic or site upgrades.
That is why Dedicated Server Admins audit our customer servers at least once a month.
During this audit, we detect possible performance bottlenecks, security loopholes and hardware issues.
This helps us to proactively resolve potential issues, rather than reacting to a downtime once an error has happened.
4. Service port blocked in firewall
Firewalls are the bedrock of server security. But if not setup right, these firewalls can cause legitimate requests to be blocked or services to fail.
For instance, in Linux servers that run Plesk automation suite, Nginx runs on port 80, and Apache runs on port 7080.
But firewalls by default block uncommon ports such as 7080, and it will result in Nginx unable to connect to Apache.
Result? 502 Bad Gateway error.
Such issues often happens when a new service is enabled (eg. caching server, Ruby, etc.) in the backend, or during a migration, or after a server upgrade.
To fix it, we look at what port each service runs on using a command like this:
# netstat -lpn
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 19785/nginx
tcp6 0 0 :::80 :::* LISTEN 19785/nginx
and if we find any service running in non-standard ports, we either change the service configuration to change it to a standard port, or edit firewall config to allow the non-standard port.
5. Web application bugs
A rare case for “502 Bad Gateway” error is application code error.
If your web server logs show a scary looing error like this, it is possible that our application code is incompatible with the server version.
[notice] child pid 27831 exit signal Segmentation fault (11)
You’ll need to inspect the software requirements of your application, and re-configure the services to match the required versions.
If you’re facing this issue right now, our Nginx experts can help you in a few minutes. Click here to open a support request. We are online 24/7.
In Summary
502 Bad Gateway in Nginx commonly occurs when Nginx runs as a reverse proxy, and is unable to connect to backend services. This can be due to service crashes, network errors, configuration issues, and more. Today we’ve seen the top 5 causes for this error, and how to fix it.
Эта ошибка имеет гораздо более простые причины, чем http 504. Она так же встречается в системах, где nginx играет роль proxy-сервера (не важно, fastcgi или http backend используется).
Nginx возвращает статус ответа HTTP-протокола 502 ровно в одном случае — он не смог достучатся до backend-сервера.
Первое, что нужно сделать — это проверить, работает ли наш backend-сервер (apache, php-fpm, unicorn, nodejs). Вот пример для php-fpm под управлением debian 8:
root@host# /etc/init.d/php5-fpm status ● php5-fpm.service - The PHP FastCGI Process Manager Loaded: loaded (/lib/systemd/system/php5-fpm.service; enabled; vendor preset: enabled) Active: inactive (dead) since Mon 2017-01-23 10:20:12 UTC; 41ms ago Process: 8011 ExecStart=/usr/sbin/php5-fpm --nodaemonize --fpm-config /etc/php5/fpm/php-fpm.conf (code=exited, status=0/SUCCESS) Process: 8005 ExecStartPre=/usr/lib/php5/php5-fpm-checkconf (code=exited, status=0/SUCCESS) Main PID: 8011 (code=exited, status=0/SUCCESS) Status: "Ready to handle connections"
Мы видим, что FPM упал и его нужно поднять:
root@host# /etc/init.d/php5-fpm start
Причины падения могут быть разные, обычно это закончившаяся память. Увидеть это можно в выводе dmesg:
root@host# dmesg ... [2358564.917301] Out of memory: Kill process 13462 (php5-fpm) score 22 or sacrifice child [2358564.917307] Killed process 13462 (php5-fpm) total-vm:2654948kB, anon-rss:92448kB, file-rss:19408kB
Сразу хочу сказать, что в PHP часто утекает память и в настройках PHP-FPM обязательно нужно указывать max_requests
max_requests = 500
А что, если FPM не упал?
Да, такая ситуация тоже возможна. Обычно она случается, когда количество одновременных запросов превышает
pm.max_children или process.max. Узнать о наступлении такой ситуации можно из лога php-fpm, а исправить увеличением количества процессов (естественно, если ресурсов для этого достаточно).
Разовые 502
Иногда встречается «разовое» появление ошибок 502 при reload или restart php-fpm, apache, unicorn и других backend-серверов. Обычно это говорит о не совсем корректной схеме деплоя (выкладке нового кода). В таком случае нужно пересматривать именно её.
HTTP 502 и nodejs
Наиболее часто эта ошибка встречается при использовании websockets. Дело в том, что для их работы требуется специальная настройка nginx: http://nginx.org/ru/docs/http/websocket.html
Так же 502 появляется при падении nodejs и перезагрузке демона. Эту проблему можно решать с помощью работы не одного процесса nodejs, а, например, трех — это обеспечит и отказоустойчивость, и возможность поочередной перезагрузки при деплое.
В виду того, что очень часто javascript-код «течет» и забивает всю память, рекомендуется использовать
Restart=always
При настройке сервиса в systemd. К сожалению, проблемы утечек памяти лежат целиком на совести разработчиков, а также очень сложны сами по себе, поэтому перезапуск «упавших» процессов и резервирование становится нормой в мире nodejs.
HTTP 502 и unicorn
Долгий запуск демона unicorn приводит к нескольким минутам ожидания и получению ошибок 502 и 504. При деплое рекомендуется использовать не «жесткую перезагрузку» unicorn, а сигнал USR2 для подмены старых рабочих процессов новыми налету.
HTTP 502 и localhost
Бывает так, что бакенд слушает исключительно ipv4 адрес 127.0.0.1, а localhost ведет на ipv6 адрес, на котором ничего нет. В таком случае рекомендуется жестко прописывать адреса — либо 127.0.0.1 для ipv4, либо соответствующий адрес для ipv6.
Итог
502-ой статус ответа говорит об однозначной невозможности открыть соединение к backend. Конечно, в сложных системах это может быть связанно с сетью, фаерволом и другими технологическими деталями конкретных систем.
| title | description | services | author | ms.service | ms.topic | ms.date | ms.author | ms.custom |
|---|---|---|---|---|---|---|---|---|
|
Troubleshoot Bad Gateway errors — Azure Application Gateway |
Learn how to troubleshoot Application Gateway Server Error: 502 — Web server received an invalid response while acting as a gateway or proxy server. |
application-gateway |
greg-lindsay |
application-gateway |
troubleshooting |
09/13/2022 |
greglin |
devx-track-azurepowershell |
Troubleshooting bad gateway errors in Application Gateway
Learn how to troubleshoot bad gateway (502) errors received when using Azure Application Gateway.
[!INCLUDE updated-for-az]
Overview
After you configure an application gateway, one of the errors that you may see is Server Error: 502 — Web server received an invalid response while acting as a gateway or proxy server. This error may happen for the following main reasons:
- NSG, UDR, or Custom DNS is blocking access to backend pool members.
- Backend VMs or instances of virtual machine scale set aren’t responding to the default health probe.
- Invalid or improper configuration of custom health probes.
- Azure Application Gateway’s backend pool isn’t configured or empty.
- None of the VMs or instances in virtual machine scale set are healthy.
- Request time-out or connectivity issues with user requests.
Network Security Group, User Defined Route, or Custom DNS issue
Cause
If access to the backend is blocked because of an NSG, UDR, or custom DNS, application gateway instances can’t reach the backend pool. This issue causes probe failures, resulting in 502 errors.
The NSG/UDR could be present either in the application gateway subnet or the subnet where the application VMs are deployed.
Similarly, the presence of a custom DNS in the VNet could also cause issues. An FQDN used for backend pool members might not resolve correctly by the user configured DNS server for the VNet.
Solution
Validate NSG, UDR, and DNS configuration by going through the following steps:
-
Check NSGs associated with the application gateway subnet. Ensure that communication to backend isn’t blocked.
-
Check UDR associated with the application gateway subnet. Ensure that the UDR isn’t directing traffic away from the backend subnet. For example, check for routing to network virtual appliances or default routes being advertised to the application gateway subnet via ExpressRoute/VPN.
$vnet = Get-AzVirtualNetwork -Name vnetName -ResourceGroupName rgName Get-AzVirtualNetworkSubnetConfig -Name appGwSubnet -VirtualNetwork $vnet -
Check effective NSG and route with the backend VM
Get-AzEffectiveNetworkSecurityGroup -NetworkInterfaceName nic1 -ResourceGroupName testrg Get-AzEffectiveRouteTable -NetworkInterfaceName nic1 -ResourceGroupName testrg -
Check presence of custom DNS in the VNet. DNS can be checked by looking at details of the VNet properties in the output.
Get-AzVirtualNetwork -Name vnetName -ResourceGroupName rgName DhcpOptions : { "DnsServers": [ "x.x.x.x" ] }
-
If present, ensure that the DNS server can resolve the backend pool member’s FQDN correctly.
Problems with default health probe
Cause
502 errors can also be frequent indicators that the default health probe can’t reach backend VMs.
When an application gateway instance is provisioned, it automatically configures a default health probe to each BackendAddressPool using properties of the BackendHttpSetting. No user input is required to set this probe. Specifically, when a load-balancing rule is configured, an association is made between a BackendHttpSetting and a BackendAddressPool. A default probe is configured for each of these associations and the application gateway starts a periodic health check connection to each instance in the BackendAddressPool at the port specified in the BackendHttpSetting element.
The following table lists the values associated with the default health probe:
| Probe property | Value | Description |
|---|---|---|
| Probe URL | http://127.0.0.1/ |
URL path |
| Interval | 30 | Probe interval in seconds |
| Time-out | 30 | Probe time-out in seconds |
| Unhealthy threshold | 3 | Probe retry count. The backend server is marked down after the consecutive probe failure count reaches the unhealthy threshold. |
Solution
- Host value of the request will be set to 127.0.0.1. Ensure that a default site is configured and is listening at 127.0.0.1.
- Protocol of the request is determined by the BackendHttpSetting protocol.
- URI Path will be set to /.
- If BackendHttpSetting specifies a port other than 80, the default site should be configured to listen at that port.
- The call to
protocol://127.0.0.1:portshould return an HTTP result code of 200. This code should be returned within the 30-second timeout period. - Ensure the configured port is open and there are no firewall rules or Azure Network Security Groups blocking incoming or outgoing traffic on the port configured.
- If Azure classic VMs or Cloud Service is used with an FQDN or a public IP, ensure that the corresponding endpoint is opened.
- If the VM is configured via Azure Resource Manager and is outside the VNet where the application gateway is deployed, a Network Security Group must be configured to allow access on the desired port.
Problems with custom health probe
Cause
Custom health probes allow additional flexibility to the default probing behavior. When you use custom probes, you can configure the probe interval, the URL, the path to test, and how many failed responses to accept before marking the backend pool instance as unhealthy.
The following additional properties are added:
| Probe property | Description |
|---|---|
| Name | Name of the probe. This name is used to refer to the probe in backend HTTP settings. |
| Protocol | Protocol used to send the probe. The probe uses the protocol defined in the backend HTTP settings |
| Host | Host name to send the probe. Applicable only when multi-site is configured on the application gateway. This is different from VM host name. |
| Path | Relative path of the probe. The valid path starts from ‘/’. The probe is sent to <protocol>://<host>:<port><path> |
| Interval | Probe interval in seconds. This is the time interval between two consecutive probes. |
| Time-out | Probe time-out in seconds. If a valid response isn’t received within this time-out period, the probe is marked as failed. |
| Unhealthy threshold | Probe retry count. The backend server is marked down after the consecutive probe failure count reaches the unhealthy threshold. |
Solution
Validate that the Custom Health Probe is configured correctly as the preceding table. In addition to the preceding troubleshooting steps, also ensure the following:
- Ensure that the probe is correctly specified as per the guide.
- If the application gateway is configured for a single site, by default the Host name should be specified as
127.0.0.1, unless otherwise configured in custom probe. - Ensure that a call to http://<host>:<port><path> returns an HTTP result code of 200.
- Ensure that Interval, Timeout, and UnhealtyThreshold are within the acceptable ranges.
- If using an HTTPS probe, make sure that the backend server doesn’t require SNI by configuring a fallback certificate on the backend server itself.
Request time-out
Cause
When a user request is received, the application gateway applies the configured rules to the request and routes it to a backend pool instance. It waits for a configurable interval of time for a response from the backend instance. By default, this interval is 20 seconds. In Application Gateway v1, if the application gateway doesn’t receive a response from backend application in this interval, the user request gets a 502 error. In Application Gateway v2, if the application gateway doesn’t receive a response from the backend application in this interval, the request will be tried against a second backend pool member. If the second request fails the user request gets a 502 error.
Solution
Application Gateway allows you to configure this setting via the BackendHttpSetting, which can be then applied to different pools. Different backend pools can have different BackendHttpSetting, and a different request time-out configured.
New-AzApplicationGatewayBackendHttpSettings -Name 'Setting01' -Port 80 -Protocol Http -CookieBasedAffinity Enabled -RequestTimeout 60
Empty BackendAddressPool
Cause
If the application gateway has no VMs or virtual machine scale set configured in the backend address pool, it can’t route any customer request and sends a bad gateway error.
Solution
Ensure that the backend address pool isn’t empty. This can be done either via PowerShell, CLI, or portal.
Get-AzApplicationGateway -Name "SampleGateway" -ResourceGroupName "ExampleResourceGroup"
The output from the preceding cmdlet should contain non-empty backend address pool. The following example shows two pools returned which are configured with an FQDN or an IP addresses for the backend VMs. The provisioning state of the BackendAddressPool must be ‘Succeeded’.
BackendAddressPoolsText:
[{
"BackendAddresses": [{
"ipAddress": "10.0.0.10",
"ipAddress": "10.0.0.11"
}],
"BackendIpConfigurations": [],
"ProvisioningState": "Succeeded",
"Name": "Pool01",
"Etag": "W/"00000000-0000-0000-0000-000000000000"",
"Id": "/subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.Network/applicationGateways/<application gateway name>/backendAddressPools/pool01"
}, {
"BackendAddresses": [{
"Fqdn": "xyx.cloudapp.net",
"Fqdn": "abc.cloudapp.net"
}],
"BackendIpConfigurations": [],
"ProvisioningState": "Succeeded",
"Name": "Pool02",
"Etag": "W/"00000000-0000-0000-0000-000000000000"",
"Id": "/subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.Network/applicationGateways/<application gateway name>/backendAddressPools/pool02"
}]
Unhealthy instances in BackendAddressPool
Cause
If all the instances of BackendAddressPool are unhealthy, then the application gateway doesn’t have any backend to route user request to. This can also be the case when backend instances are healthy but don’t have the required application deployed.
Solution
Ensure that the instances are healthy and the application is properly configured. Check if the backend instances can respond to a ping from another VM in the same VNet. If configured with a public end point, ensure a browser request to the web application is serviceable.
Next steps
If the preceding steps don’t resolve the issue, open a support ticket.