Azkaban 执行hive脚本 并从文件导入数据(org.apache.hadoop.security.AccessControlException: Permission denied...)

报异常

FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException org.apache.hadoop.security.AccessControlException: Permission denied: user=aseema, access=WRITE, inode="":hduser:supergroup:rwxr-xr-x) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

解决方法 

hadoop fs -mkdir -p /user/hive/warehouse hadoop fs -mkdir /tmp hadoop fs -chmod -R 777 /user/hive hadoop fs -chmod 777 /tmp

test.sql

use default; drop table aztest; create table aztest(id int,name string) row format delimited fields terminated by ","; load data local inpath 'b.txt' into table aztest; create table azres as select * from aztest;

 hi.job

 

重点是  load data local inpath 'b.txt''

没有local是从hdfs路径读取,加上local从本地文件读取,因为用azkaban,用local,使用相对路径即可

例如:

hi.zip

----b.txt

----hi.job

----test.sql

 

好了,这是在xshell 里面没问题

but 用azkaban跑的时候出问题了

一会permission deny 一会找不到文件,各种问题,最终look:

hadoop的用户是hdfs

hdfs.job

#hivef.job type=command command=sudo bash start.sh

 start.sh

echo "************Bash Start********************" #echo chown hdfs:hdfs * chown hdfs:hdfs b.txt chown hdfs:hdfs test.hql mv b.txt /var/lib/hadoop-hdfs mv test.hql /var/lib/hadoop-hdfs su - hdfs -c "hive -f test.hql" rm -rf /var/lib/hadoop-hdfs/* echo "************Bash End**********************"

/var/lib/hadoop-hdfs 这个目录是你不加 mv指令报错找不到文件的哪个文件夹

test.hql

create database IF not EXISTS test3 ; use test3; DROP TABLE if exists aztest; create table aztest(id int,name string) row format delimited fields terminated by ","; load data local inpath 'b.txt' into table aztest; create table azres as select * from aztest;

 b.txt

2,bb 3,cc 7,yy 9,pp

将四个文件压缩成zip,上传到azkaban,执行,搞定

报错 \r 不是shell命令什么的,用notepad将 

总而言之,一顿坑,