单点搭建HADOOP系统

    xiaoxiao2026-03-01  5

    单点搭建HADOOP系统

    安装JDK

    1.下载jdk1.6.0_20;(下载文件为jdk-6u20-linux-i586.bin);

    2.进入到jdk文件所在目录,由于运行bin文件是需要权限的,所以运行如下命令:chmod u+x jdk-6u20-linux-i586.bin,使得当前用户有权限执行该文件;

    3.在jdk文件所在目录下执行:sudo ./jdk-6u20-linux-i586.bin安装jdk;

    4.执行java -version验证jdk是否安装成功;如果安装成功会输出java的版本信息;

     

    安装Hadoop

    5.下载hadoop 0.20.2,下载后文件名是hadoop-1.0.4.tar.gz ,下载地址:http://www.apache.org/dist/hadoop/core/hadoop-0.20.2/;

    6.在hadoop文件所在目录下执行tar -zxvf hadoop-1.0.4.tar.gz,安装hadoop(所谓安装就是解压,解压出来里面有些shell脚本等);

     

    安装、配置ssh

    7.运行apt-get install openssh-server,安装ssh;

    8.配置ssh,使得可以无密码登录访问;指令如下:

        ssh-keygen -t dsa -P ''";

    执行完该指令后,在/root/.ssh目录下会出现两个文件:id_dsa和id_dsa.pub文件;

        cat ./id_dsa.pub >> authorized_keys;

    然后执行ssh localhost查看是否可以无密码登录,结果出现错误:The authenticity of host 'localhost (127.0.0.1)' can't be established. ECDSA key fingerprint is 40:2b:3c:88:b0:34:f9:cd:6d:15:b8:7b:c4:f7:02:f9. Are you sure you want to continue connecting (yes/no)?

    通过查阅资料,发现是文件权限问题,然后执行下面的命令修改文件权限:

        chmod 700 /root/.ssh     chmod 644 /root/.ssh/authorized_keys

    执行完上述命令之后,/root/.ssh文件中多了一个文件known_hosts,然后运行ssh localhost,可以无密码登录;

     

    配置Hadoop

    9.进入到hadoop安装目录,(我自己的是hadoop-1.0.4/),运行ls看到conf文件夹,这里存放的是配置相关信息;bin文件夹,存放的是可执行的文件;

    10.编辑hadoop-env.sh文件添加语句: JAVA_HOME=/opt/jdk1.6.0_20,该路径我笔者机器上jdk的安装路径,如果不清楚jdk的安装路径可以运行命令

    which java来查找安装路径;

    11.配置core-site.xml文件,文件内容如下:

    <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property>     <name>fs.default.name</name>     <value>hdfs://localhost:9000</value> </property> </configuration>

    12.配置hdfs-site.xml文件,文件内容如下:

    <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration>  <property>     <name>dfs.replication</name>     <value>1</value>  </property> </configuration>

    13.配置mapred-site.xml文件,文件内容如下:

    <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration>  <property>     <name>mapred.job.tracker</name>     <value>localhost:9001</value>  </property> </configuration>

     

    以上工作都完成之后,进入到hadoop-1.0.4/bin目录下,可以看到一些shell脚本等,通过命令sh ./start-all.sh启动hadoop,是不是可以跑啦呢^o^

    贴一下运行结果,小纪念一下,o(∩∩)o

     

    starting namenode, logging to /root/jz/hadoop-1.0.4/libexec/../logs/hadoop-root-namenode-lxh-ThinkPad-Edge.out localhost: starting datanode, logging to /root/jz/hadoop-1.0.4/libexec/../logs/hadoop-root-datanode-lxh-ThinkPad-Edge.out localhost: starting secondarynamenode, logging to /root/jz/hadoop-1.0.4/libexec/../logs/hadoop-root-secondarynamenode-lxh-ThinkPad-Edge.out starting jobtracker, logging to /root/jz/hadoop-1.0.4/libexec/../logs/hadoop-root-jobtracker-lxh-ThinkPad-Edge.out localhost: starting tasktracker, logging to /root/jz/hadoop-1.0.4/libexec/../logs/hadoop-root-tasktracker-lxh-ThinkPad-Edge.out

    最新回复(0)