Gssapi error unspecified gss failure minor code may provide more information

Hi all,

I’m trying to set up a kickstart that includes registering in the local AD.
I have managed to get it working with my trialruns using CentOS7.
Including using a dedicated KeyTab to register the machine.

  /sbin/realm join --verbose --computer-ou="...." example.com

But when I started with a RHEL7 server intended for live use the KeyTab does not work for joining the realm and when I join it manually (with -U ) I can’t log in to the new server using my AD user.

    /sbin/realm join -U sysUser@EXAMPLE.COM --verbose --computer-ou="...." example.com

I have verified that the sssd.conf and krb5.conf have the same settings.
Actually every setting I can think of is the same between the two Machines.
I tried setting SELinux to permissive mode but it did not help either.

I can use kinit to authenticate from the cli:

]$ kinit -V  myUser@EXAMPLE.COM
Using default cache: /tmp/krb5cc_1000
Using principal: myUser@EXAMPLE.COM
Password for myUser@EXAMPLE.COM:
Authenticated to Kerberos v5
]$

but the sssd service says:

]$ sudo systemctl status sssd -l
● sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/sssd.service.d
           └─journal.conf
   Active: active (running) since Mon 2018-03-05 18:22:42 CET; 1min 33s ago
 Main PID: 682 (sssd)
   CGroup: /system.slice/sssd.service
           ├─682 /usr/sbin/sssd -i -f
           ├─771 /usr/libexec/sssd/sssd_be --domain example.com --uid 0 --gid 0 --debug-to-files
           ├─924 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --debug-to-files
           └─925 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --debug-to-files

Mar 05 18:23:57 my-host@example.com sssd[be[example.com ]][771]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)
Mar 05 18:23:57 my-host@example.com sssd_be[771]: GSSAPI client step 1
Mar 05 18:23:57 my-host@example.com sssd_be[771]: GSSAPI client step 1

/var/log/sssd/sssd_example.com.log is huge even on log-level 3 but this part stands out:

(Mon Mar  5 18:22:44 2018) [sssd[be[example.com]]] [be_run_online_cb] (0x0080): Going online. Running callbacks.
(Mon Mar  5 18:22:44 2018) [sssd[be[example.com]]] [ad_sasl_log] (0x0040): SASL: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)
(Mon Mar  5 18:22:44 2018) [sssd[be[example.com]]] [sasl_bind_send] (0x0020): ldap_sasl_bind failed (-2)[Local error]
(Mon Mar  5 18:22:44 2018) [sssd[be[example.com]]] [sasl_bind_send] (0x0080): Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)]
(Mon Mar  5 18:22:44 2018) [sssd[be[example.com]]] [sdap_cli_connect_recv] (0x0040): Unable to establish connection [1432158226]: Authentication Failed

Can anyone give me some idea of where I should continue searching?

Thanks.

Regards,
//Samuel

This will be a quick post about something that was biting my ass these last few days and what was the real cause. After you read it, you are welcome to laugh at my expense. Go ahead! I deserve it!

I was working in a kerberos/ldap (linux) server and needed to debug the connection to a given client. The ldap connection uses TLS, GnuTLS specifically since the two machines were ubuntu servers, which means we also had to worry about certs. And since kerberos is in the picture, we need to configure for that. To help in solving other issues, which I should comment about later (at least those were clever problems not like this one), I was running slapd in debug mode,

/usr/sbin/slapd -d 256 -h "ldap:/// ldapi:/// ldaps:///" -g openldap -u openldap -F /etc/ldap/slapd.d

and that did help solve the other issue I had. Some of you will notice I am also running ldaps (port 636), which I really do not need since TLS should take care of the encryption thingie. But, I digress for this post, so let’s go back on topic. What I then noticed was some very problems with ldap. For instance, if I created a kerberos ticket and then tried to run ldapsearch, I would then get the following error:

root@services:~# export KRB5CCNAME=/tmp/host.tkt
root@services:~# ldapsearch -vvv
ldap_initialize(  )
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Other (e.g., implementation specific) error (80)
        additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()
root@services:~#

Here is what the server sees:

53261bde conn=1043 fd=19 ACCEPT from IP=192.168.1.181:44610 (IP=0.0.0.0:389)
53261bde conn=1043 op=0 EXT oid=1.3.6.1.4.1.1466.20037
53261bde conn=1043 op=0 STARTTLS
53261bde conn=1043 op=0 RESULT oid= err=0 text=
53261bde conn=1043 fd=19 TLS established tls_ssf=128 ssf=128
53261bde conn=1043 op=1 BIND dn="" method=163
53261bde SASL [conn=1043] Failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()
53261bde conn=1043 op=1 RESULT tag=97 err=80 text=SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()
53261bde conn=1043 op=2 UNBIND
53261bde conn=1043 fd=19 closed

Since I do not have many clever things to talk about and fill the space until the solution, how about if we talk about what some of those lines mean?

• IP=192.168.1.181:44610 (IP=0.0.0.0:389): Client 192.168.1.181 is connecting from its port 44610 to my port 389.
• oid=1.3.6.1.4.1.1466.20037: Start TLS extended request (per rfc2830).
• BIND dn=»»: anonymous if we are doing a SIMPLE bind. If we are however doing SASL bind, it is not used.
• tag=97: result from a client bind operation.

As you noticed, at least from reading the title of this post, the error line is this generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () thingie. Here is where it annoyed me to no end: what minor code? It is supposed to put some kind of message between the parenthesis, like «No principal in keytab matches desired name» or «Ticket expired». Then I would be able to search online for something. Instead, zilch. I could not find a single entry where the minor code parenthesis thingie was empty. Not very helpful today are we?

Solution

So, what was wrong? Me. User error. Do you remember how I was running slapd? Do you also remember the part about kerberos? Well, in the /etc/default/slapd (that’ll be /etc/sysconfig/ldap for you RedHat/CentOS/Fedora folks) I have defined

export KRB5_KTNAME=/etc/ldap/ldap.keytab

which means ldap knows then where the keytab containing the ldap service principal hides. Can you see where this is going? No? Let’s look again at how I am running slapd, shall we?

/usr/sbin/slapd -d 256 -h "ldap:/// ldapi:/// ldaps:///" -g openldap -u openldap -F /etc/ldap/slapd.d

As you can see, I did not pass a KRB5_KTNAME to slapd. As soon as I fed that to slapd, all was once again well in the Land of Ooo.

Log file created at: 2018/06/01 12:03:58
Running on machine: hadoop-srv01.STAT.RO
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0601 12:03:58.609741  1344 logging.cc:125] stdout will be logged to this file.
E0601 12:03:58.610020  1344 logging.cc:126] stderr will be logged to this file.
I0601 12:03:58.610366  1344 minidump.cc:231] Setting minidump size limit to 20971520.
I0601 12:03:58.610431  1344 atomicops-internals-x86.cc:98] vendor GenuineIntel  family 6  model 15  sse2 1  cmpxchg16b 1
I0601 12:03:58.619076  1344 authentication.cc:722] Using internal kerberos principal «impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO»
I0601 12:03:58.619089  1344 authentication.cc:1058] Internal communication is authenticated with Kerberos
I0601 12:03:58.619333  1344 authentication.cc:840] Waiting for Kerberos ticket for principal: impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO
I0601 12:03:58.619376  1411 authentication.cc:498] Registering impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO, keytab file /var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala.keytab
I0601 12:03:58.636188  1344 authentication.cc:842] Kerberos ticket granted to impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO
I0601 12:03:58.636248  1344 authentication.cc:722] Using external kerberos principal «impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO»
I0601 12:03:58.636255  1344 authentication.cc:1074] External communication is authenticated with Kerberos
I0601 12:03:58.636765  1344 init.cc:221] impalad version 2.10.0-cdh5.13.1 RELEASE (build 1e4b23c4eb52dac95c5be6316f49685c41783c51)
Built on Thu Nov  9 08:29:47 PST 2017
I0601 12:03:58.636775  1344 init.cc:222] Using hostname: hadoop-srv01.STAT.RO
I0601 12:03:58.637948  1344 logging.cc:161] Flags (see also /varz are on debug webserver):
—catalog_service_port=26000
—initial_hms_cnxn_timeout_s=120
—load_catalog_in_background=false
—num_metadata_loading_threads=16
—sentry_catalog_polling_frequency_s=60
—sentry_config=
—asm_module_dir=
—disable_optimization_passes=false
—dump_ir=false
—opt_module_dir=
—perf_map=false
—print_llvm_ir_instruction_count=false
—unopt_module_dir=
—abort_on_config_error=true
—be_port=22000
—be_principal=
—buffer_pool_clean_pages_limit=10%
—buffer_pool_limit=80%
—compact_catalog_topic=false
—disable_kudu=false
—disable_mem_pools=false
—enable_accept_queue_server=true
—enable_minidumps=true
—enable_process_lifetime_heap_profiling=false
—enable_stats_extrapolation=false
—heap_profile_dir=
—hostname=hadoop-srv01.STAT.RO
—inc_stats_size_limit_bytes=209715200
—keytab_file=/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala.keytab
—krb5_conf=
—krb5_debug_file=
—kudu_operation_timeout_ms=180000
—load_auth_to_local_rules=false
—max_minidumps=9
—mem_limit=49893343232
—min_buffer_size=65536
—minidump_path=/var/log/impala-minidumps/impalad
—minidump_size_limit_hint_kb=20480
—principal=impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO
—redaction_rules_file=
—local_library_dir=/var/lib/impala/udfs
—max_audit_event_log_files=0
—max_log_files=10
—memory_maintenance_sleep_time_ms=10000
—pause_monitor_sleep_time_ms=500
—pause_monitor_warn_threshold_ms=10000
—log_filename=impalad
—redirect_stdout_stderr=true
—data_source_batch_size=1024
—exchg_node_buffer_size_bytes=10485760
—enable_quadratic_probing=true
—skip_lzo_version_check=false
—parquet_min_filter_reject_ratio=0.10000000000000001
—runtime_filter_wait_time_ms=1000
—suppress_unknown_disk_id_warnings=false
—max_row_batches=0
—kudu_max_row_batches=0
—kudu_read_mode=READ_LATEST
—kudu_scanner_keep_alive_period_sec=15
—pick_only_leaders_for_tests=false
—kudu_mutation_buffer_size=10485760
—kudu_sink_mem_required=20971520
—convert_legacy_hive_parquet_utc_timestamps=false
—max_page_header_size=8388608
—accepted_cnxn_queue_depth=10000
—enable_ldap_auth=false
—internal_principals_whitelist=hdfs
—kerberos_reinit_interval=60
—ldap_allow_anonymous_binds=false
—ldap_baseDN=
—ldap_bind_pattern=
—ldap_ca_certificate=
—ldap_domain=
—ldap_manual_config=false
—ldap_passwords_in_clear_ok=false
—ldap_tls=false
—ldap_uri=
—sasl_path=
—concurrent_scratch_ios_per_device=2
—madvise_huge_pages=true
—mmap_buffers=false
—insert_inherit_permissions=false
—datastream_sender_timeout_ms=120000
—adls_read_chunk_size=131072
—use_hdfs_pread=false
—max_cached_file_handles=0
—max_free_io_buffers=128
—num_adls_io_threads=16
—num_disks=0
—num_io_threads_per_rotational_disk=0
—num_io_threads_per_solid_state_disk=0
—num_remote_hdfs_io_threads=8
—num_s3_io_threads=16
—num_threads_per_disk=0
—read_size=8388608
—unused_file_handle_timeout_sec=270
—backend_client_connection_num_retries=3
—backend_client_rpc_timeout_ms=300000
—catalog_client_connection_num_retries=3
—catalog_client_rpc_timeout_ms=0
—catalog_service_host=hadoop-mgmt01.STAT.RO
—coordinator_rpc_threads=12
—disable_admission_control=false
—enable_webserver=true
—num_hdfs_worker_threads=16
—state_store_host=hadoop-mgmt01.STAT.RO
—state_store_subscriber_port=23000
—status_report_interval=5
—s3a_access_key_cmd=
—s3a_secret_key_cmd=
—log_mem_usage_interval=0
—max_filter_error_rate=0.75
—num_threads_per_core=3
—use_local_tz_for_unix_timestamp_conversions=false
—allow_multiple_scratch_dirs_per_device=false
—disk_spill_encryption=false
—scratch_dirs=/data/data01/impala/impalad,/data/data02/impala/impalad,/data/data03/impala/impalad,/data/data04/impala/impalad,/data/data05/impala/impalad,/data/data06/impala/impalad,/data/data07/impala/impalad,/data/data08/impala/impalad,/data/data09/impala/impalad,/data/data10/impala/impalad,/data/data11/impala/impalad,/data/data12/impala/impalad,/data/data13/impala/impalad
—queue_wait_timeout_ms=60000
—default_pool_max_queued=200
—default_pool_max_requests=-1
—default_pool_mem_limit=
—disable_pool_max_requests=false
—disable_pool_mem_limits=false
—fair_scheduler_allocation_path=/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/fair-scheduler.xml
—llama_site_path=/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/llama-site.xml
—require_username=false
—authorization_policy_file=
—authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
—authorized_proxy_user_config=
—authorized_proxy_user_config_delimiter=,
—kudu_master_hosts=
—server_name=
—abort_on_failed_audit_event=false
—abort_on_failed_lineage_event=true
—audit_event_log_dir=/var/log/impalad/audit
—beeswax_port=21000
—cancellation_thread_pool_size=5
—default_query_options=
—fe_service_threads=64
—hs2_port=21050
—idle_query_timeout=0
—idle_session_timeout=0
—is_coordinator=true
—is_executor=true
—lineage_event_log_dir=/var/log/impalad/lineage
—log_query_to_file=true
—max_audit_event_log_file_size=5000
—max_lineage_log_file_size=5000
—max_profile_log_file_size=5000
—max_profile_log_files=10
—max_result_cache_size=100000
—profile_log_dir=
—query_log_size=25
—ssl_cipher_list=
—ssl_client_ca_certificate=
—ssl_minimum_version=tlsv1
—ssl_private_key=
—ssl_private_key_password_cmd=
—ssl_server_certificate=
—statestore_subscriber_cnxn_attempts=10
—statestore_subscriber_cnxn_retry_interval_ms=3000
—statestore_subscriber_timeout_seconds=30
—state_store_port=24000
—statestore_heartbeat_frequency_ms=1000
—statestore_heartbeat_tcp_timeout_seconds=3
—statestore_max_missed_heartbeats=10
—statestore_num_heartbeat_threads=10
—statestore_num_update_threads=10
—statestore_update_frequency_ms=2000
—statestore_update_tcp_timeout_seconds=300
—force_lowercase_usernames=false
—num_cores=0
—web_log_bytes=1048576
—non_impala_java_vlog=0
—periodic_counter_update_period_ms=500
—enable_webserver_doc_root=true
—webserver_authentication_domain=
—webserver_certificate_file=
—webserver_doc_root=/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala
—webserver_interface=
—webserver_password_file=
—webserver_port=25000
—webserver_private_key_file=
—webserver_private_key_password_cmd=
—webserver_x_frame_options=DENY
—flagfile=/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/impalad_flags
—fromenv=
—tryfromenv=
—undefok=
—tab_completion_columns=80
—tab_completion_word=
—help=false
—helpfull=false
—helpmatch=
—helpon=
—helppackage=false
—helpshort=false
—helpxml=false
—version=false
—alsologtoemail=
—alsologtostderr=false
—colorlogtostderr=false
—drop_log_memory=true
—log_backtrace_at=
—log_dir=/var/log/impalad
—log_link=
—log_prefix=true
—logbuflevel=0
—logbufsecs=30
—logemaillevel=999
—logmailer=/bin/mail
—logtostderr=false
—max_log_size=200
—minloglevel=0
—stderrthreshold=4
—stop_logging_if_full_disk=false
—symbolize_stacktrace=true
—v=1
—vmodule=
I0601 12:03:58.638094  1344 init.cc:227] Cpu Info:
  Model: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
  Cores: 40
  Max Possible Cores: 40
  L1 Cache: 32.00 KB (Line: 64.00 B)
  L2 Cache: 256.00 KB (Line: 64.00 B)
  L3 Cache: 25.00 MB (Line: 64.00 B)
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    avx
    avx2
  Numa Nodes: 2
  Numa Nodes of Cores: 0->0 | 1->0 | 2->0 | 3->0 | 4->0 | 5->0 | 6->0 | 7->0 | 8->0 | 9->0 | 10->1 | 11->1 | 12->1 | 13->1 | 14->1 | 15->1 | 16->1 | 17->1 | 18->1 | 19->1 | 20->0 | 21->0 | 22->0 | 23->0 | 24->0 | 25->0 | 26->0 | 27->0 | 28->0 | 29->0 | 30->1 | 31->1 | 32->1 | 33->1 | 34->1 | 35->1 | 36->1 | 37->1 | 38->1 | 39->1 |
I0601 12:03:58.638119  1344 init.cc:228] Disk Info:
  Num disks 15:
    sdc (rotational=true)
    sde (rotational=true)
    sdf (rotational=true)
    sdh (rotational=true)
    sdb (rotational=true)
    sdd (rotational=true)
    sdj (rotational=true)
    sdi (rotational=true)
    sdg (rotational=true)
    sda (rotational=false)
    sdk (rotational=true)
    sdm (rotational=true)
    sdn (rotational=true)
    sdl (rotational=true)
    dm- (rotational=true)

I0601 12:03:58.638208  1344 init.cc:229] Physical Memory: 126.04 GB
Transparent Huge Pages Config:
  enabled: always madvise [never]
  defrag: always madvise [never]
  khugepaged defrag: [yes] no
I0601 12:03:58.638216  1344 init.cc:230] OS version: Linux version 2.6.32-696.16.1.el6.x86_64 (mockbuild@c1bl.rdu2.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC) ) #1 SMP Wed Nov 15 16:51:15 UTC 2017
Clock: clocksource: ‘tsc’, clockid_t: CLOCK_MONOTONIC
I0601 12:03:58.638221  1344 init.cc:231] Process ID: 1344
I0601 12:04:00.055625  1344 llvm-codegen.cc:136] CPU class for runtime code generation: broadwell
I0601 12:04:00.055665  1344 llvm-codegen.cc:139] CPU flags for runtime code generation: -sse4a,-avx512bw,+cx16,-tbm,+xsave,-fma4,-avx512vl,+prfchw,+bmi2,+adx,-xsavec,+fsgsbase,+avx,-avx512cd,-avx512pf,+rtm,+popcnt,+fma,+bmi,+aes,+rdrnd,-xsaves,+sse4.1,+sse4.2,+avx2,-avx512er,+sse,+lzcnt,+pclmul,-avx512f,+f16c,+ssse3,+mmx,-pku,+cmov,-xop,+rdseed,+movbe,+hle,+xsaveopt,-sha,+sse2,+sse3,-avx512dq
I0601 12:04:00.749959  1344 GlogAppender.java:137] Logging (re)initialized. Impala: VLOG, All other: INFO
I0601 12:04:00.752050  1344 JniFrontend.java:139] Authorization is ‘DISABLED’.
I0601 12:04:00.752928  1344 JniFrontend.java:141] Java Input arguments:
-Djava.library.path=/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/bin/../lib/impala/lib
Java System properties:
java.vm.version:24.65-b04
awt.toolkit:sun.awt.X11.XToolkit
java.vendor.url:http://java.oracle.com/
sun.jnu.encoding:UTF-8
java.vm.info:mixed mode
sun.boot.library.path:/usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64
java.home:/usr/java/jdk1.7.0_67-cloudera/jre
user.dir:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD
java.vm.name:Java HotSpot(TM) 64-Bit Server VM
sun.cpu.isalist:
java.awt.graphicsenv:sun.awt.X11GraphicsEnvironment
sun.os.patch.level:unknown
java.endorsed.dirs:/usr/java/jdk1.7.0_67-cloudera/jre/lib/endorsed
sun.management.compiler:HotSpot 64-Bit Tiered Compilers
user.home:/var/lib/impala
java.runtime.name:Java(TM) SE Runtime Environment
java.io.tmpdir:/tmp
java.awt.printerjob:sun.print.PSPrinterJob
java.version:1.7.0_67
file.encoding.pkg:sun.io
java.library.path:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/bin/../lib/impala/lib
file.separator:/
java.vendor.url.bug:http://bugreport.sun.com/bugreport/
java.specification.vendor:Oracle Corporation
file.encoding:UTF-8
line.separator:

java.vm.specification.version:1.7
java.vm.specification.vendor:Oracle Corporation
user.timezone:Europe/Tiraspol
os.name:Linux
java.vm.vendor:Oracle Corporation
path.separator::
java.ext.dirs:/usr/java/jdk1.7.0_67-cloudera/jre/lib/ext:/usr/java/packages/lib/ext
java.class.path:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/var/lib/impala/*.jar:/usr/share/java/mysql-connector-java.jar:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/hadoop-conf:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/hive-conf:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/hbase-conf:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libthrift-0.9.0.jar::/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ST4-4.0.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ant-1.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ant-1.9.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ant-contrib-1.0b3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ant-launcher-1.9.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/antlr-runtime-3.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/aopalliance-1.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/apache-log4j-extras-1.2.17.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/apacheds-i18n-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/apacheds-jdbm1-2.0.0-M2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/api-asn1-api-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/api-util-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/asm-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/asm-commons-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/asm-tree-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/async-1.3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/audience-annotations-0.4.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/avro.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/aws-java-sdk-bundle-1.11.134.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/azure-data-lake-store-sdk-2.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/bonecp-0.7.1.RELEASE.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/calcite-avatica-1.0.0-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/calcite-core-1.0.0-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/calcite-linq4j-1.0.0-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-beanutils-1.8.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-beanutils-core-1.8.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-codec-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-compiler-2.7.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-configuration-1.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-daemon-1.0.13.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-dbcp-1.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-digester-1.8.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-el-1.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-httpclient-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-io-2.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-lang3-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-logging-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-math-2.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-math3-3.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-net-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-pool-1.5.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-pool2-2.4.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/core-3.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-client-2.7.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-framework-2.7.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-recipes-2.7.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-test-2.11.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-x-discovery-2.11.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/datanucleus-api-jdo-3.2.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/datanucleus-core-3.2.12.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/datanucleus-rdbms-3.2.12.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/derby-10.10.2.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/eigenbase-properties-1.1.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/findbugs-annotations-1.3.9-1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/flatbuffers-java-1.6.0.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/geronimo-annotation_1.0_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/geronimo-jaspic_1.0_spec-1.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/geronimo-jta_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/groovy-all-2.4.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/gson-2.2.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/guice-3.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/guice-servlet-3.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-archives.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-azure-datalake.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-hdfs.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-mapreduce-client-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-mapreduce-client-core.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-api.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-client.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-nodemanager.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-web-proxy.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hbase-annotations.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hbase-client.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hbase-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hbase-protocol.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/high-scale-lib-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-ant.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-beeline.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-cli.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-exec.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-hbase-handler.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-hcatalog-core.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-hcatalog-server-extensions.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-metastore.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-serde.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-service.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-shims-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-shims-scheduler.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-shims.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hsqldb-1.8.0.10.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/htrace-core-3.0.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/htrace-core-3.2.0-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/htrace-core4-4.0.1-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/httpclient-4.2.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/httpcore-4.2.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/impala-data-source-api-1.0-SNAPSHOT.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/impala-frontend-0.1-SNAPSHOT.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ivy-2.4.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-annotations-2.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-core-2.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-core-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-databind-2.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-jaxrs-1.8.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-mapper-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-xc-1.8.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jamon-runtime-2.3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/janino-2.7.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jasper-compiler-5.5.23.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jasper-runtime-5.5.23.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/java-cup-0.11-a-czt02-cdh.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/java-cup-runtime-0.11-a-czt01-cdh.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/java-xmlbuilder-0.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javassist-3.18.1-GA.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javax.inject-1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javax.jdo-3.2.0-m3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javax.json-1.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javax.servlet-2.5.0.v201103041518.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jaxb-api-2.2.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jaxb-impl-2.2.3-1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jcodings-1.0.8.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jdo-api-3.0.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-client-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-core-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-guice-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-json-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-server-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-servlet-1.14.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jets3t-0.9.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jettison-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-6.1.26.cloudera.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-all-7.6.0.v20120127.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-continuation-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-http-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-io-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-security-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-server-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-servlet-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-util-6.1.26.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-util-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jline-2.12.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jms-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/joda-time-2.8.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/joni-2.1.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jpam-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jsch-0.1.42.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/json-simple-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jsp-api-2.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jsr305-3.0.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jta-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/kudu-client-1.5.0-cdh5.13.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/leveldbjni-all-1.8.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libfb303-0.9.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libgcc_s.so.1:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libhadoop.so:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libhadoop.so.1.0.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libhdfs.so:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libhdfs.so.0.0.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libkudu_client.so.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libkudu_client.so.0.1.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libstdc++.so.6:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libstdc++.so.6.0.20:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libthrift-0.9.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/logredactor-1.0.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/mail-1.4.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-core-2.2.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-core-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-healthchecks-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-json-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-jvm-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-servlets-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/mockito-all-1.9.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/netty-3.10.5.Final.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/netty-all-4.0.23.Final.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/okhttp-2.4.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/okio-1.4.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/opencsv-2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/oro-2.0.8.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/paranamer-2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/postgresql-9.0-801.jdbc4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-binding-hive-conf.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-binding-hive-follower.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-binding-hive.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-core-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-core-model-db.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-core-model-kafka.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-core-model-search.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-hdfs-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-policy-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-policy-db.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-policy-kafka.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-policy-search.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-provider-cache.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-provider-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-provider-db-sh.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-provider-file.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/shiro-core-1.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/slf4j-log4j12-1.7.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/stax-api-1.0-2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/stax-api-1.0.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/stringtemplate-3.2.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/super-csv-2.2.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/transaction-api-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/velocity-1.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/xercesImpl-2.9.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/xml-apis-1.3.04.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/xmlenc-0.52.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/xz-1.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/zookeeper.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/openssl::/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libcrypto.so:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libcrypto.so.1.0.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libssl.so:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libssl.so.1.0.0::/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
sun.arch.data.model:64
os.version:2.6.32-696.16.1.el6.x86_64
java.specification.name:Java Platform API Specification
sun.io.unicode.encoding:UnicodeLittle
user.language:en
user.name:impala
os.arch:amd64
java.runtime.version:1.7.0_67-b01
sun.boot.class.path:/usr/java/jdk1.7.0_67-cloudera/jre/lib/resources.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/rt.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/sunrsasign.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/jsse.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/jce.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/charsets.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/jfr.jar:/usr/java/jdk1.7.0_67-cloudera/jre/classes
user.country:US
java.class.version:51.0
java.vendor:Oracle Corporation
java.vm.specification.name:Java Virtual Machine Specification
sun.cpu.endian:little
java.specification.version:1.7
I0601 12:04:01.343031  1344 AllocationFileLoaderService.java:212] Loading allocation file /var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/fair-scheduler.xml
I0601 12:04:01.384876  1344 impala-server.cc:1215] Default query options:TQueryOptions {
  01: abort_on_error (bool) = false,
  02: max_errors (i32) = 100,
  03: disable_codegen (bool) = false,
  04: batch_size (i32) = 0,
  05: num_nodes (i32) = 0,
  06: max_scan_range_length (i64) = 0,
  07: num_scanner_threads (i32) = 0,
  08: max_io_buffers (i32) = 0,
  09: allow_unsupported_formats (bool) = false,
  10: default_order_by_limit (i64) = -1,
  11: debug_action (string) = «»,
  12: mem_limit (i64) = 0,
  13: abort_on_default_limit_exceeded (bool) = false,
  15: hbase_caching (i32) = 0,
  16: hbase_cache_blocks (bool) = false,
  17: parquet_file_size (i64) = 0,
  18: explain_level (i32) = 1,
  19: sync_ddl (bool) = false,
  23: disable_cached_reads (bool) = false,
  24: disable_outermost_topn (bool) = false,
  25: rm_initial_mem (i64) = 0,
  26: query_timeout_s (i32) = 0,
  28: appx_count_distinct (bool) = false,
  29: disable_unsafe_spills (bool) = false,
  31: exec_single_node_rows_threshold (i32) = 100,
  32: optimize_partition_key_scans (bool) = false,
  33: replica_preference (i32) = 0,
  34: schedule_random_replica (bool) = false,
  35: scan_node_codegen_threshold (i64) = 1800000,
  36: disable_streaming_preaggregations (bool) = false,
  37: runtime_filter_mode (i32) = 2,
  38: runtime_bloom_filter_size (i32) = 1048576,
  39: runtime_filter_wait_time_ms (i32) = 0,
  40: disable_row_runtime_filtering (bool) = false,
  41: max_num_runtime_filters (i32) = 10,
  42: parquet_annotate_strings_utf8 (bool) = false,
  43: parquet_fallback_schema_resolution (i32) = 0,
  45: s3_skip_insert_staging (bool) = true,
  46: runtime_filter_min_size (i32) = 1048576,
  47: runtime_filter_max_size (i32) = 16777216,
  48: prefetch_mode (i32) = 1,
  49: strict_mode (bool) = false,
  50: scratch_limit (i64) = -1,
  51: enable_expr_rewrites (bool) = true,
  52: decimal_v2 (bool) = false,
  53: parquet_dictionary_filtering (bool) = true,
  54: parquet_array_resolution (i32) = 2,
  55: parquet_read_statistics (bool) = true,
  56: default_join_distribution_mode (i32) = 0,
  57: disable_codegen_rows_threshold (i32) = 50000,
  58: default_spillable_buffer_size (i64) = 2097152,
  59: min_spillable_buffer_size (i64) = 65536,
  60: max_row_size (i64) = 524288,
}
I0601 12:04:01.385177  1460 RequestPoolService.java:150] Loading Llama configuration: /var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/llama-site.xml
W0601 12:04:01.491346  1344 UserGroupInformation.java:1920] PriviledgedActionException as:impala (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0601 12:04:01.493019  1344 Client.java:713] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0601 12:04:01.493314  1344 UserGroupInformation.java:1920] PriviledgedActionException as:impala (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
I0601 12:04:01.506008  1344 status.cc:122] Could not read the root directory at hdfs://hadoop-mgmt01.STAT.RO:8020. Error was:
Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: «hadoop-srv01.STAT.RO/192.168.128.11»; destination host is: «hadoop-mgmt01.STAT.RO»:8020;
    @           0x83e789  impala::Status::Status()
    @           0xa79206  impala::Frontend::ValidateSettings()
    @           0xaa0124  impala::ImpalaServer::ImpalaServer()
    @           0xaa18d6  impala::CreateImpalaServer()
    @           0xb186d7  ImpaladMain()
    @           0x7d9423  main
    @     0x7f3814795d1d  __libc_start_main
    @           0x80c2dd  (unknown)
E0601 12:04:01.506031  1344 impala-server.cc:282] Could not read the root directory at hdfs://hadoop-mgmt01.STAT.RO:8020. Error was:
Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: «hadoop-srv01.STAT.RO/192.168.128.11»; destination host is: «hadoop-mgmt01.STAT.RO»:8020;
E0601 12:04:01.506062  1344 impala-server.cc:285] Aborting Impala Server startup due to improper configuration. Impalad exiting.

Сеть не настроена или настроена не полностью:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
     address i.i.i.i
     netmask 255.255.m.m
     gateway g.g.g.g

Не указан gateway в настройках сети клиента.

Не заполнен /etc/ald/ald.conf

Не указан gateway в настройках сети клиента.

Не указан gateway в настройках сети клиента.

Некорректно настроены имя и ip адрес, например в /etc/hosts отсутствует длинное имя машин:

127.0.0.1 localhost
192.168.1.1  myserver
192.168.1.2	 client

Ошибка может быть вызвана применением антивируса или изменением правил iptables.

Некорректно настроены имя и ip адрес, например в /etc/hosts (разрешение имен может быть настроено и с помощью сервера DNS) следует указать ip, длинное и короткое имя и исключить запись «127.0.1.1 myserver»:

127.0.0.1 localhost
192.168.1.1 myserver.example.ru myserver
192.168.1.2	client.example.ru client

В /etc/ald/ald.conf не указаны длинное имя машины и домен:

DOMAIN=.example.ru
 
SERVER=myserver.example.ru

• При вводе клиента (ald-client join), ошибка(345) возникает из-за несовпадения времени на машинах.

Утилита hostname должна выдавать короткое имя. После внесения изменений в файл /etc/hostname необходимо перезагрузить машину.

Разрешение имен настроено не верно, перепутаны местами длинное и короткое имя, пример:

127.0.0.1 localhost
192.168.1.1  myserver myserver.example.ru
192.168.1.2  client client.example.ru

После добавления astra-admin в ALLOWED_LOCAL_GROUPS и выполнения ald-init commit-config снять права администратора домена, применить, установить права администратора домена, применить.

• При выполнении: ald-client commit-config

— ОШИБКА:
Ошибка OpenLDAP при запросе ‘cn=client.ru,ou=hosts,dc=ru (objectClass=x-ald-host-object)’ — Can’tcontact LDAP server в ALDLdapConnection.cpp:213(Search)
—

На клиенте в файле /etc/hosts не внесены данные о резервном сервере.