mapred-site.xml文件的内容:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
尽管我设置了yarn.app.mapreduce.am.env和其他参数,但我仍然收到“找不到或加载主类org.apache.hadoop.mapreduce.v2.app.MRAppMaster”的错误。
我正在尝试远程运行map reduce程序,在此过程中,hadoop已安装在Linux机器上,我正在从Windows机器上运行它。
以下是我的作业配置设置。public class WordCount {
public static void main(String[] args)
throws IOException, ClassNotFoundException, InterruptedException {
//
UserGroupInformation ugi = UserGroupInformation.createRemoteUser("admin");
ugi.doAs(new PrivilegedExceptionAction<Void>() {
public Void run() throws Exception {
try {
Configuration configuration = new Configuration();
configuration.set("yarn.resourcemanager.address", "192.168.33.75:50001"); // see step 3
configuration.set("mapreduce.framework.name", "yarn");
// configuration.set("yarn.app.mapreduce.am.env",
// "HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0");
// configuration.set("mapreduce.map.env", "HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0");
// configuration.set("mapreduce.reduce.env",
// "HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0");
configuration.set("fs.defaultFS", "hdfs://192.168.33.75:54310"); // see step 2
configuration.set("mapreduce.app-submission.cross-platform", "true");
configuration.set("mapred.remote.os", "Linux");
configuration.set("yarn.application.classpath",
"$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:"
+ "$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:"
+ "$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:"
+ "$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*");
Job job = Job.getInstance(configuration);
job.setJarByClass(WordCount.class); // use this when uploaded the Jar to the server and
// running the job directly and locally on the server
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(MapForWordCount.class);
job.setReducerClass(ReduceForWordCount.class);
Path input = new Path("/user/admin/wordCountInput.txt");
Path output = new Path("/user/admin/output");
FileInputFormat.addInputPath(job, input);
FileOutputFormat.setOutputPath(job, output);
System.exit(job.waitForCompletion(true) ? 0 : 1);
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
});
}
请帮帮我,我已经被这个问题困扰了6天。非常感谢您的帮助。
Hadoop版本:3.1.0
/home/admin/hadoop-3.1.0
符号链接到/home/admin/hadoop
可能会很方便...当您升级hadoop时,更改符号链接,但不必在所有配置文件中更改版本号。 - dannyman