An error occurred while trying to connect to the java server - Исправление ошибок и поиск оптимальных решений проблем

Good afternoon,

In the last two days occurs many connection problems to the Java server. It´s a little bit uncommon because the error occurs not always, only sometimes…

I am using PySpark combined with Jupyter Notebook. Everything is running on a VM instance in the Google Cloud. I am using this one in Google Cloud:

custom (8 vCPUs, 200 GB)

These are the other settings:

conf = pyspark.SparkConf().setAppName("App")
conf = (conf.setMaster('local[*]')
        .set('spark.executor.memory', '180G')
        .set('spark.driver.memory', '180G')
        .set('spark.driver.maxResultSize', '180G'))

sc = pyspark.SparkContext(conf=conf)
sq = pyspark.sql.SQLContext(sc)

I trained a Random Forest Model and made predictions:

model = rf.fit(train)
predictions = model.transform(test)

Afterwards I created the ROC-Curve and compute the AUC-value.

Then I wanted to see the confusion matrix:

confusion_mat = metrics.confusionMatrix().toArray()
print(confusion_mat_train_rf)

And now the error occurs:

    Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 652, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/dist-packages/pyspark/accumulators.py", line 235, in handle
    num_updates = read_int(self.rfile)
  File "/usr/local/lib/python2.7/dist-packages/pyspark/serializers.py", line 577, in read_int
    raise EOFError
EOFError
ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/py4j/java_gateway.py", line 883, in send_command
    response = connection.send_command(command)
  File "/usr/local/lib/python2.7/dist-packages/py4j/java_gateway.py", line 1040, in send_command
    "Error while receiving", e, proto.ERROR_ON_RECEIVE)
Py4JNetworkError: Error while receiving
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:39543)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/py4j/java_gateway.py", line 963, in start
    self.socket.connect((self.address, self.port))
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

Here is the output from the console:

OpenJDK 64-Bit Server VM warning
: INFO: os::commit_memory(0x00007f4998300000, 603979776, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 603979776 bytes for committing reserved memory.

Logfile:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 603979776 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2643), pid=2377, tid=0x00007f1c94fac700
#
# JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
# Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  S Y S T E M  ---------------

OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"

uname:Linux 4.13.0-1008-gcp #11-Ubuntu SMP Thu Jan 25 11:08:44 UTC 2018 x86_64
libc:glibc 2.23 NPTL 2.23 
rlimit: STACK 8192k, CORE 0k, NPROC 805983, NOFILE 1048576, AS infinity
load average:7.69 4.51 3.57

/proc/meminfo:
MemTotal:       206348252 kB
MemFree:         1298460 kB
MemAvailable:     250308 kB
Buffers:            6812 kB
Cached:           438232 kB
SwapCached:            0 kB
Active:         203906416 kB
Inactive:         339540 kB
Active(anon):   203804300 kB
Inactive(anon):     8392 kB
Active(file):     102116 kB
Inactive(file):   331148 kB
Unevictable:        3652 kB
Mlocked:            3652 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              4688 kB
Writeback:             0 kB
AnonPages:      203805168 kB
Mapped:            23076 kB
Shmem:              8776 kB
Slab:             114476 kB
SReclaimable:      50640 kB
SUnreclaim:        63836 kB
KernelStack:        4752 kB
PageTables:       404292 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    103174124 kB
Committed_AS:   205956256 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       71628 kB
DirectMap2M:     4122624 kB
DirectMap1G:    207618048 kB


CPU:total 8 (initial active 8) (4 cores per cpu, 2 threads per core) family 6 model 85 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx

Does anyone have any idea what the problem might be and how i can solve this? I am desperate.

// I think the Java Runtime Environment has not enough memory to continue… But what can i do?

Thank you very much!

Источник

Hey,

I am still facing the same issue of connection failure. I am attaching the log of the issue.

ERROR:root:Exception while sending command.
Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1159, in send_command
raise Py4JNetworkError(«Answer from Java side is empty»)
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 985, in send_command
response = connection.send_command(command)
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1164, in send_command
«Error while receiving», e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
Traceback (most recent call last):
File «/usr/lib/python3.5/socketserver.py», line 313, in _handle_request_noblock
self.process_request(request, client_address)
File «/usr/lib/python3.5/socketserver.py», line 341, in process_request
self.finish_request(request, client_address)
File «/usr/lib/python3.5/socketserver.py», line 354, in finish_request
self.RequestHandlerClass(request, client_address, self)
File «/usr/lib/python3.5/socketserver.py», line 681, in init
self.handle()
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/accumulators.py», line 269, in handle
poll(accum_updates)
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/accumulators.py», line 241, in poll
if func():
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/accumulators.py», line 245, in accum_updates
num_updates = read_int(self.rfile)
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/serializers.py», line 717, in read_int
raise EOFError
EOFError
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/sql/dataframe.py», line 534, in collect
sock_info = self._jdf.collectToPython()
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1257, in call
answer, self.gateway_client, self.target_id, self.name)
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/sql/utils.py», line 63, in deco
return f(*a, **kw)
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/protocol.py», line 336, in get_return_value
format(target_id, «.», name))
py4j.protocol.Py4JError: An error occurred while calling o1501.collectToPython

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1159, in send_command
raise Py4JNetworkError(«Answer from Java side is empty»)
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/IPython/core/interactiveshell.py», line 3326, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File «», line 19, in
print(df[‘warn_type2_total’,].head(3))
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/sql/dataframe.py», line 1253, in head
return self.take(n)
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/sql/dataframe.py», line 572, in take
return self.limit(num).collect()
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/sql/dataframe.py», line 534, in collect
sock_info = self._jdf.collectToPython()
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/traceback_utils.py», line 78, in exit
self._context._jsc.setCallSite(None)
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1257, in call
answer, self.gateway_client, self.target_id, self.name)
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/sql/utils.py», line 63, in deco
return f(*a, **kw)
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/protocol.py», line 336, in get_return_value
format(target_id, «.», name))
py4j.protocol.Py4JError: An error occurred while calling o22.setCallSite

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/IPython/core/interactiveshell.py», line 2040, in showtraceback
stb = value.render_traceback()
AttributeError: ‘Py4JError’ object has no attribute ‘render_traceback‘

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:44098)
Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/sql/dataframe.py», line 534, in collect
sock_info = self._jdf.collectToPython()
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1257, in call
answer, self.gateway_client, self.target_id, self.name)
File «/home/ubuntu/venv/lib/python3.5/site-packages/pyspark/sql/utils.py», line 63, in deco
return f(*a, **kw)
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/protocol.py», line 336, in get_return_value
format(target_id, «.», name))
py4j.protocol.Py4JError: An error occurred while calling o1501.collectToPython

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused

Exception happened during processing of request from (‘127.0.0.1’, 39990)

Py4JError Traceback (most recent call last)
~/venv/lib/python3.5/site-packages/pyspark/sql/dataframe.py in collect(self)
533 with SCCallSiteSync(self._sc) as css:
—> 534 sock_info = self._jdf.collectToPython()
535 return list(_load_from_socket(sock_info, BatchedSerializer(PickleSerializer())))

~/venv/lib/python3.5/site-packages/py4j/java_gateway.py in call(self, *args)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258

~/venv/lib/python3.5/site-packages/pyspark/sql/utils.py in deco(*a, **kw)
62 try:
—> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:

~/venv/lib/python3.5/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
335 «An error occurred while calling {0}{1}{2}».
—> 336 format(target_id, «.», name))
337 else:

Py4JError: An error occurred while calling o1501.collectToPython

During handling of the above exception, another exception occurred:

Py4JError Traceback (most recent call last)
in
17
18 print(df[‘warn_type1_total’,].head(3))
—> 19 print(df[‘warn_type2_total’,].head(3))

~/venv/lib/python3.5/site-packages/pyspark/sql/dataframe.py in head(self, n)
1251 rs = self.head(1)
1252 return rs[0] if rs else None
-> 1253 return self.take(n)
1254
1255 @ignore_unicode_prefix

~/venv/lib/python3.5/site-packages/pyspark/sql/dataframe.py in take(self, num)
570 [Row(age=2, name=u’Alice’), Row(age=5, name=u’Bob’)]
571 «»»
—> 572 return self.limit(num).collect()
573
574 @SInCE(1.3)

~/venv/lib/python3.5/site-packages/pyspark/sql/dataframe.py in collect(self)
532 «»»
533 with SCCallSiteSync(self._sc) as css:
—> 534 sock_info = self._jdf.collectToPython()
535 return list(_load_from_socket(sock_info, BatchedSerializer(PickleSerializer())))
536

~/venv/lib/python3.5/site-packages/pyspark/traceback_utils.py in exit(self, type, value, tb)
76 SCCallSiteSync._spark_stack_depth -= 1
77 if SCCallSiteSync._spark_stack_depth == 0:
—> 78 self._context._jsc.setCallSite(None)

~/venv/lib/python3.5/site-packages/py4j/java_gateway.py in call(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

~/venv/lib/python3.5/site-packages/pyspark/sql/utils.py in deco(*a, **kw)
61 def deco(*a, **kw):
62 try:
—> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()

~/venv/lib/python3.5/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
334 raise Py4JError(
335 «An error occurred while calling {0}{1}{2}».
—> 336 format(target_id, «.», name))
337 else:
338 type = answer[1]

Py4JError: An error occurred while calling o22.setCallSite

Can anyone help me solving this issue?
Thanks

Источник

I am trying to create spark context object with the following commands on pyspark:

ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35918)
Traceback (most recent call last):
File «/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py», line 963, in start
self.socket.connect((self.address, self.port))
File «/usr/local/lib/python2.7/socket.py», line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
Traceback (most recent call last):
File «<stdin>», line 1, in <module>
File «/home/gamedev/spark-2.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/dataframe.py», line 419, in coalesce
File «/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py», line 1131, in _call_
answer = self.gateway_client.send_command(command)
File «/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py», line 881, in send_command
connection = self._get_connection()
File «/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py», line 829, in _get_connection
connection = self._create_connection()
File «/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py», line 835, in _create_connection
connection.start()
File «/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py», line 970, in start
raise Py4JNetworkError(msg, e)
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:35918)

———————————————
I try spark version 2.0.0, 2.0.1, 2.0.2
no problem 2.0.0
but 2.0.1, 2.0.2 occur

python code
——
…..
df = spark.read.json(‘hdfs://big_big_4000000.json’)

json_log = []
for log in df.collect():
jj = {}
try:
for f in log._fields_:
if f == ‘I_LogDes’:
if log[f] is not None:
log_des_json = json.loads(log[f])
for jf in log_des_json:
json_key = add_2(jf)
if json_key in jj:
json_key = ‘%s_2’ % json_key
jj[json_key] = typeIntStr(log_des_json[jf])
else:
jj[remove_i(f)] = typeIntStr(log[f])
json_log.append(jj)
except:
print log

!!! here error occur
df = spark.read.json(spark.sparkContext.parallelize(json_log))

Источник

#python #python-3.x #apache-spark #pyspark #py4j

Вопрос:

Когда я выполнял метод наименьших квадратов для выполнения факторизации матрицы, я столкнулся с ошибкой с Java-сервером в Spark, я не знаю, почему это происходит, ниже приведено сообщение об ошибке.

 Exception happened during processing of request from ('127.0.0.1', 61711)
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socketserver.py", line 720, in __init__
    self.handle()
  File "/usr/local/opt/apache-spark/libexec/python/pyspark/accumulators.py", line 262, in handle
    poll(accum_updates)
  File "/usr/local/opt/apache-spark/libexec/python/pyspark/accumulators.py", line 235, in poll
    if func():
  File "/usr/local/opt/apache-spark/libexec/python/pyspark/accumulators.py", line 239, in accum_updates
    num_updates = read_int(self.rfile)
  File "/usr/local/opt/apache-spark/libexec/python/pyspark/serializers.py", line 564, in read_int
    raise EOFError
EOFError
----------------------------------------
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:61698)
Traceback (most recent call last):
  File "/Users/zhengyangzhang/Library/Python/3.8/lib/python/site-packages/IPython/core/interactiveshell.py", line 3417, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-deb1baad55f8>", line 15, in <module>
    MSE = ratesAndPreds.map(lambda r: (r[1][0] - r[1][1])**2).mean()
  File "/usr/local/opt/apache-spark/libexec/python/pyspark/rdd.py", line 1386, in mean
    return self.stats().mean()
  File "/usr/local/opt/apache-spark/libexec/python/pyspark/rdd.py", line 1245, in stats
    return self.mapPartitions(lambda i: [StatCounter(i)]).reduce(redFunc)
  File "/usr/local/opt/apache-spark/libexec/python/pyspark/rdd.py", line 997, in reduce
    vals = self.mapPartitions(func).collect()
  File "/usr/local/opt/apache-spark/libexec/python/pyspark/rdd.py", line 949, in collect
    sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
  File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, in __call__
    return_value = get_return_value(
  File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: <unprintable Py4JJavaError object>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zhengyangzhang/Library/Python/3.8/lib/python/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'Py4JJavaError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1207, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1033, in send_command
    response = connection.send_command(command)
  File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1211, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while receiving

Приведенное ниже сообщение повторялось сотни раз в моем блокноте Jupyter.

 
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 977, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1115, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 61] Connection refused

Я задаю этот вопрос, потому что я просто не понимаю, почему это происходит, если кто-то может объяснить мне, почему это происходит, и предоставить мне возможное решение этой проблемы, я был бы очень признателен.

Ниже приведен список моих пакетов импорта и настроек:

 import findspark
findspark.init()
from pyspark import SparkContext
from pyspark.mllib.recommendation import ALS, MatrixFactorizationModel, Rating

Я прочитал в Интернете, что пакет findspark может помочь решить эту проблему, поэтому я завариваю apache-spark и загружаю этот пакет, но все равно возникают проблемы.

Источник

Welcome to the Cloudera Community

Your Hybrid Data Community. With over 95,300 members and 20,700 solutions, you’ve come to the right place!

cancel

Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Show only

|

Search instead for

Did you mean:

Advanced Search

Cloudera Community
Archived

This Content was Archived

Try searching for related content

Источник

Petr Jezek

unread,

Jul 20, 2017, 3:18:08 AM7/20/17

to Py4J Support and Comments

When I’m connecting to a remote host I’m getting following error: py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (xx.xx.xx.xx:25333)) (where xx.xx.xx.xx is the IP address I don’t want to share). When I’m running gateway server on a localhost I’m able to connect without issues. In both cases I’m using default ports without any changing.

I’m fairly sure that firewall or something like that is not a problem because the host is a common ubuntu installation wihtout any IP tables rules configured.

Thanks

Petr

Barthelemy Dagenais

unread,

Jul 20, 2017, 11:37:22 AM7/20/17

to Petr Jezek, Py4J Support and Comments

Barthelemy Dagenais

unread,

Jul 20, 2017, 11:37:52 AM7/20/17

to Petr Jezek, Py4J Support and Comments

err, I meant JavaGateway (Python)

Petr Jezek

unread,

Jul 20, 2017, 8:34:24 PM7/20/17

to Py4J Support and Comments

Sure.

>>> from py4j.java_gateway import JavaGateway

>>> from py4j.java_gateway import GatewayParameters

>>> gateway = JavaGateway(gateway_parameters=GatewayParameters(address='<IP>’))

>>> res = gateway.executeQuery()

Traceback (most recent call last):

File «<stdin>», line 1, in <module>

File «/home/petr-jezek/.local/lib/python2.7/site-packages/py4j/java_gateway.py», line 1158, in __call__

answer = self.gateway_client.send_command(command)

File «/home/petr-jezek/.local/lib/python2.7/site-packages/py4j/java_gateway.py», line 906, in send_command

connection = self._get_connection()

File «/home/petr-jezek/.local/lib/python2.7/site-packages/py4j/java_gateway.py», line 854, in _get_connection

connection = self._create_connection()

File «/home/petr-jezek/.local/lib/python2.7/site-packages/py4j/java_gateway.py», line 860, in _create_connection

connection.start()

File «/home/petr-jezek/.local/lib/python2.7/site-packages/py4j/java_gateway.py», line 997, in start

raise Py4JNetworkError(msg, e)

py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (<IP>:25333)

>>>

Where <IP> is the IP of the server

And the server has been run like:

GatewayServer gatewayServer = new GatewayServer(facade);
gatewayServer.start();

where facade is an interface with the executeQuery method.

And as I said, where java server and python client are run both from localhost it works.

Petr

Dne čtvrtek 20. července 2017 1:37:52 UTC-7 Barthelemy Dagenais napsal(a):

Petr Jezek

unread,

Jul 20, 2017, 8:41:31 PM7/20/17

to Py4J Support and Comments

Dne středa 19. července 2017 17:18:08 UTC-7 Petr Jezek napsal(a):

Petr Jezek

unread,

Jul 20, 2017, 8:42:20 PM7/20/17

to Py4J Support and Comments

Dne středa 19. července 2017 17:18:08 UTC-7 Petr Jezek napsal(a):

Petr Jezek

unread,

Jul 20, 2017, 8:42:44 PM7/20/17

to Py4J Support and Comments

Dne středa 19. července 2017 17:18:08 UTC-7 Petr Jezek napsal(a):

Barthelemy Dagenais

unread,

Jul 20, 2017, 9:41:25 PM7/20/17

to Petr Jezek, Py4J Support and Comments

Petr Jezek

unread,

Jul 21, 2017, 1:51:57 AM7/21/17

to Py4J Support and Comments

Thanks,

It helped. If I used the different constructor, the python code starts working.

Petr

Dne čtvrtek 20. července 2017 11:41:25 UTC-7 Barthelemy Dagenais napsal(a):

Источник

Здравствуйте, я работал с Pyspark, впервые реализуя проект анализа настроений с использованием пакета ML. Код работал хорошо, но вдруг он показывает ошибку, упомянутую выше:

   ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:50532)
Traceback (most recent call last):
  File "C:optsparkspark-2.3.0-bin-hadoop2.7pythonlibpy4j-0.10.6-src.zippy4jjava_gateway.py", line 852, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:optsparkspark-2.3.0-bin-hadoop2.7pythonlibpy4j-0.10.6-src.zippy4jjava_gateway.py", line 990, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [WinError 10061] Aucune connexion n’a pu être établie car l’ordinateur cible l’a expressément refusée

Кто-нибудь может помочь, пожалуйста. Вот полное описание ошибки?

5 ответов

Просто перезагрузите ноутбук, если вы используете Jupyter nootbook. Если нет, то просто перезапустите pyspark. Это должно решить проблему. Это происходит из-за того, что вы используете слишком много сборок или из-за какой-либо другой проблемы, связанной с памятью.

Andy_101
12 Фев 2021 в 22:00

Добавьте больше ресурсов в Spark. Например, если вы работаете в локальном режиме, должно быть достаточно следующей конфигурации:

spark = SparkSession.builder 
.appName('app_name') 
.master('local[*]') 
.config('spark.sql.execution.arrow.pyspark.enabled', True) 
.config('spark.sql.session.timeZone', 'UTC') 
.config('spark.driver.memory','32G') 
.config('spark.ui.showConsoleProgress', True) 
.config('spark.sql.repl.eagerEval.enabled', True) 
.getOrCreate()

Pedro Muñoz
4 Фев 2021 в 11:40

Я столкнулся с этой ошибкой при попытке использовать PySpark в контейнере Docker. В моем случае ошибка возникла из-за того, что я назначил Spark больше ресурсов, чем имел доступ Docker.

Neelotpal Shukla
24 Апр 2020 в 02:39

Я столкнулся с той же проблемой, работая над colab. Я завершил текущий сеанс и снова подключился. Это сработало для меня!

Keerthi Reddy
28 Янв 2021 в 15:56

Возможно порт spark UI уже занят, возможно перед этой ошибкой есть другие ошибки.

Возможно, это может вам помочь: https://stackoverflow.com/questions/32820087/spark-multiple-spark-submit-in-parallel

spark-submit --conf spark.ui.port=5051

Majid Hajibaba
7 Июн 2021 в 13:05

Источник

Traceback (most recent call last): File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1067, in start self.socket.connect((self.address, self.port)) ConnectionRefusedError: [Errno 111] Connection refused

Exception happened during processing of request from (‘127.0.0.1’, 39990)

Вопрос:

Welcome to the Cloudera Community

Petr Jezek

Barthelemy Dagenais

Barthelemy Dagenais

Petr Jezek

Petr Jezek

Petr Jezek

Petr Jezek

Barthelemy Dagenais

Petr Jezek

5 ответов

Traceback (most recent call last):
File «/home/ubuntu/venv/lib/python3.5/site-packages/py4j/java_gateway.py», line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused