Execution error return code 1 from org apache hadoop hive ql exec tez teztask

2016-12-19 04:43:41,466 ERROR [main]: exec.Task (TezTask.java:execute(212)) - Failed to execute tez graph.
org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1481976368061_0017 to YARN : Application application_1481976368061_0017 submitted by user root to unknown queue: agent
    at org.apache.tez.client.TezClient.start(TezClient.java:413)
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
    at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271)
    at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1481976368061_0017 to YARN : Application application_1481976368061_0017 submitted by user root to unknown queue: agent
    at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271)
    at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
    at org.apache.tez.client.TezClient.start(TezClient.java:408)
    ... 22 more
2016-12-19 04:43:41,466 INFO  [main]: hooks.ATSHook (ATSHook.java:<init>(90)) - Created ATS Hook
2016-12-19 04:43:41,466 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver>
2016-12-19 04:43:41,470 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1482122621466 end=1482122621470 duration=4 from=org.apache.hadoop.hive.ql.Driver>
2016-12-19 04:43:41,471 ERROR [main]: ql.Driver (SessionState.java:printError(932)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
2016-12-19 04:43:41,471 INFO  [main]: ql.Driver (Driver.java:execute(1621)) - Resetting the caller context to 
2016-12-19 04:43:41,471 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=Driver.execute start=1482122620489 end=1482122621471 duration=982 from=org.apache.hadoop.hive.ql.Driver>
2016-12-19 04:43:41,471 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2016-12-19 04:43:41,471 INFO  [ATS Logger 0]: hooks.ATSHook (ATSHook.java:createPostHookEvent(193)) - Received post-hook notification for :root_20161219044340_094281c1-c0f2-4cc9-83a5-962b067937f7
2016-12-19 04:43:41,488 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=releaseLocks start=1482122621471 end=1482122621488 duration=17 from=org.apache.hadoop.hive.ql.Driver>
2016-12-19 04:43:41,505 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(135)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2016-12-19 04:43:41,505 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(162)) - </PERFLOG method=releaseLocks start=1482122621505 end=1482122621505 duration=0 from=org.apache.hadoop.hive.ql.Driver>

Beeline Error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)

Issue:

Some time ago, I assisted a customer who was trying to do a select count(*) via beeline and failed with:

1
2
3

[hadoop@ip-10-10-9-226 ~]$ beeline -u jdbc:hive2://nn-emr.sysint.dxone.local:10000/default --silent=true --outputformat=csv2 -e "select count(*) from basetables_rms.rms_site"
19/04/26 06:41:15 [main]: WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not support autoCommit=false.
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)

When reproducing this I found a jira: https://issues.apache.org/jira/browse/HIVE-14631 which related to the same issue and the workaround was to switch your execution engine to mapreduce. By doing that, it worked, but wanted a better resolution for the customer.

Debugging:

When setting enabling debugging, I found that the error is related to permissions:

1
2
3
4
5
6
7
8

$ beeline  -u jdbc:hive2://172.31.31.247:10000/default --silent=false --outputformat=csv2 -e "select count(*) from testdb.users"
Connecting to jdbc:hive2://172.31.31.247:10000/default
Connected to: Apache Hive (version 2.1.1-amzn-0)
Driver: Hive JDBC (version 2.1.1-amzn-0)
19/04/26 10:24:01 [main]: WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not support autoCommit=false.
...
ERROR : Failed to execute tez graph.
org.apache.hadoop.security.AccessControlException: Permission denied: user=anonymous, access=WRITE, inode="/user/anonymous":hdfs:hadoop:drwxr-xr-x

So it seems that when the client (anonymous) is trying to copy the hive execution jar to is home path in HDFS, in this case (/home/anonymous/.hiveJars/) it fails due to permissions.

Resolution:

By passing the hadoop user, I was able to get the expected results:

1
2
3
4
5
6

$ beeline -n hadoop -u jdbc:hive2://172.31.31.247:10000/default --silent=false --outputformat=csv2 -e "select count(*) from testdb.users"
INFO  : Completed executing command(queryId=hive_20190426103246_33253d86-3ebc-462f-a5a1-f01877dd00a8); Time taken: 17.08 seconds
INFO  : OK
c0
1
1 row selected (17.282 seconds)

Listing the mentioned jar:

1
2
3

$ hdfs dfs -ls /user/hadoop/.hiveJars/
Found 1 items
-rw-r--r--   1 hadoop hadoop   32447131 2019-04-26 09:51 /user/hadoop/.hiveJars/hive-exec-2.1.1-amzn-0-ac46be4721493d9e62fd1b132ecee3d20fd283680edbc0cfa9809c656a493469.jar

Hope this might help someone facing the same issue

I have a Hive server using Tez that I can connect correctly:

import pyhive
from pyhive import hive

cursor = hive.connect(connect_host, port=10000, configuration={'job.queue.name': 'myqueue'}).cursor()
cursor.execute('use db_20160111') # works fine
cursor.execute('show databases') # works fine
cursor.execute('SELECT cgi FROM db_20160111.useful_result') # crashes

with the following error:

OperationalError: TExecuteStatementResp(status=TStatus(errorCode=1, errorMessage='Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask', sqlState='08S01', infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask:29:28', 'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:314', 'org.apache.hive.service.cli.operation.SQLOperation:runQuery:SQLOperation.java:146', 'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:173', 'org.apache.hive.service.cli.operation.Operation:run:Operation.java:256', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:376', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatement:HiveSessionImpl.java:357', 'sun.reflect.GeneratedMethodAccessor308:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', 'java.lang.reflect.Method:invoke:Method.java:606', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:79', 'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:37', 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:64', 'java.security.AccessController:doPrivileged:AccessController.java:-2', 'javax.security.auth.Subject:doAs:Subject.java:415', 'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1628', 'org.apache.hadoop.hive.shims.HadoopShimsSecure:doAs:HadoopShimsSecure.java:536', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:60', 'com.sun.proxy.$Proxy23:executeStatement::-1', 'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:234', 'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:401', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1313', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1298', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:206', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:745'], statusCode=3), operationHandle=None)

Thanks!