Java.lang.RuntimeException: 无法实例化org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Question

我在Ubuntu 14.0上安装了Hadoop 2.7.1和apache-hive-1.2.1版本。

1. 为什么会出现这个错误？
2. 需要安装任何元存储吗？
3. 当我们在终端键入hive命令时，xml是如何被内部调用的，这些xml的流程是什么？
4. 是否需要其他配置？

当我在Ubuntu 14.0终端上编写hive命令时，它会抛出以下异常。

 $ hive

    Logging initialized using configuration in jar:file:/usr/local/hive/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
    Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:520)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
    Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
        ... 8 more
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:426)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
        ... 14 more
    Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
    NestedThrowables:
    java.lang.reflect.InvocationTargetException
        at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:520)
        at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
        at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394)
        at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291)
        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:57)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:571)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:624)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
        ... 19 more
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:426)
        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)
        at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:282)
        at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:240)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:286)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:426)
        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
        at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
        at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
        ... 48 more
    Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:259)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:131)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:85)
        ... 66 more
    Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
        at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)
        at org.datanucleus.store.rdbms.connectionpool.BoneCPConnectionPoolFactory.createConnectionPool(BoneCPConnectionPoolFactory.java:54)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:238)
        ... 68 more

为了避免上述错误，我创建了带有以下内容的hive-site.xml文件：

 <configuration>

      <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>/home/local/hive-metastore-dir/warehouse</value>

      </property>

    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://localhost:3306/hivedb?createDatabaseIfNotExist=true</value>
    </property>
    <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>user</value>
    </property>
    <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>password</value>
    </property>

    <configuration>

在~/.bashrc文件中提供环境变量，但错误仍然存在。

#HIVE home directory configuration
export HIVE_HOME=/usr/local/hive/apache-hive-1.2.1-bin
export PATH="$PATH:$HIVE_HOME/bin"

- Arti Nalawade

---

我也在这个链接中回答了这个编程问题：https://dev59.com/k2Eh5IYBdhLWcg3wMA0t#51499009 - Hamid

你的hive-site.xml文件中，闭合标签</configuration>缺少了斜线/。 - moshfiqur

Answer 1

启动hive metastore服务对我有用。首先，为hive metastore设置数据库：

 $ hive --service metastore 

`https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/validate_installation.html`

其次，运行以下命令：

 $ schematool -dbType mysql -initSchema  
 $ schematool -dbType mysql -info

https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool

https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool

- Ravi Vyas

---

在我的情况下，运行“hive --service metastore &”并等待5分钟就足以修复Hive。 - 42n4

也可以在本地模式下使用 derby。只需将上述命令中的 mysql 替换为 derby 即可。 例如： schematool -dbType derby -initSchema - WestCoastProjects

Answer 2

我进行了下面的修改，现在我能够无错误地启动Hive Shell：

1. ~/.bashrc

在bashrc文件的结尾添加以下环境变量：sudo gedit ~/.bashrc

#Java Home directory configuration
export JAVA_HOME="/usr/lib/jvm/java-9-oracle"
export PATH="$PATH:$JAVA_HOME/bin"

# Hadoop home directory configuration
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

export HIVE_HOME=/usr/lib/hive
export PATH=$PATH:$HIVE_HOME/bin

2. hive-site.xml

你需要在Hive的conf目录下创建这个文件(hive-site.xml)，并添加以下细节。

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://localhost/metastore?createDatabaseIfNotExist=true</value>
</property>


<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>root</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>root</value>
</property>

<property>
  <name>datanucleus.autoCreateSchema</name>
  <value>true</value>
</property>

<property>
  <name>datanucleus.fixedDatastore</name>
  <value>true</value>
</property>

<property>
 <name>datanucleus.autoCreateTables</name>
 <value>True</value>
 </property>

</configuration>

3. 您还需要将jar文件(mysql-connector-java-5.1.28.jar)放在Hive的lib目录中。

4. 在Ubuntu上启动Hive Shell所需的以下安装：

1. MySql
2. Hadoop
3. Hive
4. Java

5. 执行部分：

1. 启动所有Hadoop服务：start-all.sh

2. 输入jps命令检查所有Hadoop服务是否已经启动：jps

3. 输入hive命令进入Hive shell：hive

- Arti Nalawade

---

我找不到解决方案，也不知道我哪里出了错... 当我在hive>show databases;提示符下运行命令时，我收到错误消息“FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient”。请帮我看看我哪里做错了。 - Shyam Gupta

使用以下命令在Centos上获取jar包： wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.28/mysql-connector-java-5.1.28.jar - Saurav Sahu

不建议将Hadoop的sbin文件夹导出到PATH路径中。 - undefined

Answer 3

如果你只是在本地模式下玩耍，你可以删除元数据存储数据库并重新安装它：

rm -rf metastore_db/
$HIVE_HOME/bin/schematool -initSchema -dbType derby

- Ajeden

---

明智的回答。在尝试任何复杂的回答之前，有其他人试过这个吗？ - Mark Maxwell

Answer 4

在我的情况下，当我尝试时

$ hive --service metastore 

我遇到了如下问题：

MetaException(message: 在metastore中找不到版本信息。)

MySQL中缺少metastore所需的必要表格，请手动创建这些表格，并重新启动hive metastore。

cd $HIVE_HOME/scripts/metastore/upgrade/mysql/ 

< Login into MySQL > 

mysql> drop database IF EXISTS <metastore db name>; 
mysql> create database <metastore db name>; 
mysql> use <metastore db name>; 
mysql> source hive-schema-2.x.x.mysql.sql; 

元数据存储数据库名称应与hive-site.xml文件连接属性标签中提到的数据库名称相匹配。

hive-schema-2.x.x.mysql.sql文件取决于当前目录中可用的版本。尽量选择最新的，因为它还包含许多旧架构文件。

现在尝试执行hive --service metastore 如果一切顺利，那么可以直接从终端启动hive。

>hive

我希望上面的答案能满足您的需求。

- vijayraj34

---

hive --service metastore & 就足以修复hive了。 - 42n4

@42n4 你的意思是我只需要在结尾加上'&'吗？ - vijayraj34

它曾经运行了一段时间，但是我不得不重新启动主服务器才能从所有错误中恢复。 - 42n4

Answer 5

在调试模式下运行Hive：

hive -hiveconf hive.root.logger=DEBUG,console

然后执行以下命令：

show tables

可以找到实际问题。

- Vignesh

Answer 6

您只需要实例化模式，以下命令也可以完成相同的操作。我这样做了，能够运行Hive查询而不会抛出错误：ERROR:Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient。

cd $HIVE_HOME
mv metastore_db metastore_db_bkup
schematool -initSchema -dbType derby
bin/hive

现在执行你的查询：

hive> show databases;

- krishna kumar mishra

Answer 7

我在Hive MetaStore中使用了MySQL数据库。请按照以下步骤进行：

• 在hive-site.xml中，metastore应该正确配置。

<property>
 <name>javax.jdo.option.ConnectionURL</name> 
 <value>jdbc:mysql://localhost/metastorecreateDatabaseIfNotExist=true&amp;useSSL=false</value>
</property>

• 进入mysql: mysql -u hduser -p
• 然后运行drop database metastore
• 退出MySQL并执行schematool -initSchema dbType mysql

现在错误将不再出现。

- arunava maiti

Answer 8

在堆栈跟踪的中间，被“反射”垃圾所淹没，您可以找到根本原因：

指定的数据存储驱动程序（“com.mysql.jdbc.Driver”）未在CLASSPATH中找到。请检查您的CLASSPATH规范和驱动程序名称。

- Samson Scharfrichter

---

嗨，Samson Scharfrichter，根据建议，我已经提供了CLASSPATH规范，现在DatastoreDriverNotFoundException问题已经解决了。 但是仍然存在异常：Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient。 有什么建议吗？ - Arti Nalawade

Answer 9

"最初的回答"可能是你的Hive元数据存储存在不一致！我也遇到过这种情况。

首先，我运行了：

 $ schematool -dbType mysql -initSchema  

最初的回答：

然后我找到了这个信息

错误：重复的键名'PCS_STATS_IDX'（state= 42000，code=1061）org.apache.hadoop.hive.metastore.HiveMetaException：模式初始化失败！Metastore状态将不一致！

然后我运行了...

 $ schematool -dbType mysql -info

这是个错误提示： 需要执行以下操作：

- 删除名为hive_db的mysql数据库 - 运行schematool -dbType mysql -initSchema以初始化元数据

- libin

Answer 10

这可能是由于它缺乏与Hive元存储的连接所致，我的Hive元存储存储在Mysql中，因此我需要访问Mysql，所以我在我的build.sbt中添加了一个依赖项。

libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.38"

问题已经解决！

- Doom