博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
oozie4.0.0安装
阅读量:5034 次
发布时间:2019-06-12

本文共 9696 字,大约阅读时间需要 32 分钟。

安装完hadoop后便可安装oozie运行自己的工作流:

1、下载oozie压缩包,oozie-4.0.0-cdh5.0.0.tar.gz,下载地址

2、下载ext-2.2.zip:

3、下载tomcat并解压

4、下载maven,(下载的oozie是已经编译好的,如果是未编译的需要用maven先编译一下才能安装)

5、解压oozie到安装目录,解压并设置环境变量如下:

export MAVEN_HOME=/export/servers/apache-maven-3.0.5

export TOMCAT_HOME=/export/servers/apache-tomcat-6.0.26

export OOZIE_HOME=/export/servers/oozie-4.0.0-cdh5.0.0

export OOZIE_CONFIG=/export/servers/oozie-4.0.0-cdh5.0.0/conf
export PATH=$JAVA_HOME/bin:JRE_HOME/bin:$HADOOP_HOME/bin:$MAVEN_HOME/bin:$OOZIE_HOME/bin:$TOMCAT_HOME/bin:$PATH

使环境变量生效:source /etc/profile

6、修改oozie配置文件,conf目录结构如下:

action-conf目录下只有一个hive.xml文件,修改内容如下:

<configuration>

<!-- An example of setting default properties for Hive action.
This could be useful with Hadoop versions that have deprecated
HADOOP_HOME that Hive still relies on.

<property>

<name>hadoop.bin.path</name>
<value>/export/servers/hadoop-2.2.0/bin/hadoop</value>
</property>

<property>

<name>hadoop.config.dir</name>
<value>/export/servers/hadoop-2.2.0/etc/hadoop</value>
</property>
-->
</configuration>

hadoop-conf/core-site.xml:

<configuration>

<property>

<name>mapreduce.jobtracker.kerberos.principal</name>
<value>mapred/_HOST@LOCALREALM</value>
</property>

<property>

<name>yarn.resourcemanager.principal</name>
<value>yarn/_HOST@LOCALREALM</value>
</property>

<property>

<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/_HOST@LOCALREALM</value>
</property>

<property>

<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

</configuration>

hadoop-config.xml文件内容与hadoop-conf/core-site.xml文件内容一样,无需修改。

oozie-default.xml该文件改动主要有两处:

1、<property>

<name>oozie.services</name>
<value>
org.apache.oozie.service.SchedulerService,
org.apache.oozie.service.InstrumentationService,
org.apache.oozie.service.CallableQueueService,
org.apache.oozie.service.UUIDService,
org.apache.oozie.service.ELService,
org.apache.oozie.service.AuthorizationService,
org.apache.oozie.service.UserGroupInformationService,
org.apache.oozie.service.HadoopAccessorService,
org.apache.oozie.service.URIHandlerService,
org.apache.oozie.service.MemoryLocksService,
org.apache.oozie.service.DagXLogInfoService,
org.apache.oozie.service.SchemaService,
org.apache.oozie.service.LiteWorkflowAppService,
org.apache.oozie.service.JPAService,
org.apache.oozie.service.StoreService,
org.apache.oozie.service.CoordinatorStoreService,
org.apache.oozie.service.SLAStoreService,
org.apache.oozie.service.DBLiteWorkflowStoreService,
org.apache.oozie.service.CallbackService,
org.apache.oozie.service.ActionService,
org.apache.oozie.service.ShareLibService,
org.apache.oozie.service.ActionCheckerService,
org.apache.oozie.service.RecoveryService,
org.apache.oozie.service.PurgeService,
org.apache.oozie.service.CoordinatorEngineService,
org.apache.oozie.service.BundleEngineService,
org.apache.oozie.service.DagEngineService,
org.apache.oozie.service.CoordMaterializeTriggerService,
org.apache.oozie.service.StatusTransitService,
org.apache.oozie.service.PauseTransitService,
org.apache.oozie.service.GroupsService,
org.apache.oozie.service.ProxyUserService,
org.apache.oozie.service.XLogStreamingService,
org.apache.oozie.service.JobsConcurrencyService
</value>
<description>
All services to be created and managed by Oozie Services singleton.
Class names must be separated by commas.
</description>
</property>

将该节点的org.apache.oozie.service.JobsConcurrencyService类提至第一行,如下:

<property>

<name>oozie.services</name>
<value>
org.apache.oozie.service.JobsConcurrencyService,
org.apache.oozie.service.SchedulerService,

。。。。

2、去掉下面节点,(其实去不去无所谓,根据自己的实际应用来)

<property>

<name>oozie.service.coord.check.maximum.frequency</name>
<value>true</value>
<description>
When true, Oozie will reject any coordinators with a frequency faster than 5 minutes. It is not recommended to disable
this check or submit coordinators with frequencies faster than 5 minutes: doing so can cause unintended behavior and
additional system stress.
</description>
</property>

oozie-site.xml,修改的地方主要有以下几点:

1、<property>

<name>oozie.service.ActionService.executor.ext.classes</name>
<value>
org.apache.oozie.action.email.EmailActionExecutor,
org.apache.oozie.action.hadoop.HiveActionExecutor,
org.apache.oozie.action.hadoop.ShellActionExecutor,
org.apache.oozie.action.hadoop.SqoopActionExecutor,
org.apache.oozie.action.hadoop.DistcpActionExecutor
</value>
</property>该节点修改成如下,添加几项内容:

<property>

<name>oozie.subworkflow.classpath.inheritance</name>
<value>true</value>
</property>
<property>
<name>oozie.servlet.CallbackServlet.max.data.len</name>
<value>1048576</value>
</property>

<property>

<name>oozie.service.ActionService.executor.ext.classes</name>
<value>
org.apache.oozie.action.email.EmailActionExecutor,
org.apache.oozie.action.hadoop.HiveActionExecutor,
org.apache.oozie.action.hadoop.ShellActionExecutor,
org.apache.oozie.action.hadoop.SqoopActionExecutor,
org.apache.oozie.action.hadoop.DistcpActionExecutor
</value>
</property>

2、

<property>

<name>oozie.service.JPAService.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>
<description>
JDBC driver class.
</description>
</property>

<property>

<name>oozie.service.JPAService.jdbc.url</name>
<value>jdbc:mysql://192.168.157.92:3358/oozie4</value>
<description>
JDBC URL.
</description>
</property>

<property>

<name>oozie.service.JPAService.jdbc.username</name>
<value>root</value>
<description>
DB user name.
</description>
</property>

<property>

<name>oozie.service.JPAService.jdbc.password</name>
<value>123456</value>
<description>
DB user password.

IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,

if empty Configuration assumes it is NULL.
</description>
</property>

这几个节点的作用是:oozie有一个默认的derby数据库,是用来存储oozie节点的相关信息的,如果想用自己的mysql数据库,可按照上面例子配置

3、

<property>

<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/export/servers/hadoop-2.2.0/etc/hadoop</value>
<description>
Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
the relevant Hadoop *-site.xml files. If the path is relative is looked within
the Oozie configuration directory; though the path can be absolute (i.e. to point
to Hadoop client conf/ directories in the local filesystem.
</description>
</property>

该节点是设置hadoop的配置文件目录

4、

<!-- Proxyuser Configuration -->

<property>

<name>oozie.service.ProxyUserService.proxyuser.#USER#.hosts</name>
<value>*</value>
<description>
List of hosts the '#USER#' user is allowed to perform 'doAs'
operations.

The '#USER#' must be replaced with the username o the user who is

allowed to perform 'doAs' operations.

The value can be the '*' wildcard or a list of hostnames.

For multiple users copy this property and replace the user name

in the property name.
</description>
</property>

<property>

<name>oozie.service.ProxyUserService.proxyuser.#USER#.groups</name>
<value>*</value>
<description>
List of groups the '#USER#' user is allowed to impersonate users
from to perform 'doAs' operations.

The '#USER#' must be replaced with the username o the user who is

allowed to perform 'doAs' operations.

The value can be the '*' wildcard or a list of groups.

For multiple users copy this property and replace the user name

in the property name.
</description>
</property>

将该两个节点的注释去掉。

因为用到了mysql数据库,所以需要将mysql的jar包mysql-connector-java-5.1.20.jar拷贝到oozie的lib目录及libtools目录下

至此,oozie的配置文件修改完毕,下面做一些oozie启动前的准备工作 :

1、进入到mysql数据库,创建在oozie-site.xml文件中指定的数据库oozie:

    create database oozie;    (创建名称为oozie的数据库)

    grant all privileges on oozie.* to 'root'@'localhost' identified by '123456';    (设置oozie数据库的访问全选,创建用户名为oozie,密码为oozie的用户)
    grant all privileges on oozie.* to 'root'@'%' identified by '123456';    (设置oozie数据库的访问权限)
    FLUSH PRIVILEGES;

2、在$OOZIE_HOME/bin目录下执行以下命令,生成创建数据库表的脚本:

 sh ooziedb.sh create -sqlfile oozie.sql

3、执行数据库脚本,生成相关数据库表:

 sh oozie-setup.sh db create -run  -sqlfile oozie.sql

至此,数据库配置完毕。

4、生成oozie.war包:

执行如下命令生成oozie.war包,还是在bin目录下执行:

sh addtowar.sh -inputwar $OOZIE_HOME/oozie.war -outputwar $OOZIE_HOME/oozie-server/webapps/oozie.war -hadoop 2.2.0 $HADOOP_HOME -extjs ext-2.2.zip

5、生成的war包可能没有带mysql-connector-java-5.1.20.jar包,所以需要将该jar包也加到war包中去,否则后面启动oozie时会报错。

6、在$OOZIE_HOME/bin目录下执行(不知道该步骤有啥用)

sh oozie-setup.sh sharelib create -fs hdfs://hadoop-master:8020 -locallib $OOZIE_HOME/oozie-sharelib-4.0.0-cdh5.2.0-yarn.tar.gz

(hadoop2多hdfs集群,hdfs://cluster1是core-site.xml中defaultFs名称:)sh oozie-setup.sh sharelib create -fs hdfs://cluster1 -locallib $OOZIE_HOME/oozie-sharelib-4.0.0-cdh5.0.0-beta-2-yarn.tar.gz

6、启动oozie:

前台运行oozie:

sh oozied.sh run

后台运行oozie:

sh oozied.sh start

启动后看看http://hadoop-master:11000/oozie如下:

7、运行ooize配置:

运行oozie工作流的目录基本机构如下:

8、将上面目录文件上传至hdfs目录下,如/user/root/oozie/workflow/oozieTest目录下

执行oozie命令如下,可以将命令直接放到一个sh文件中,下次直接sh执行即可:

run_oozie.sh:

oozie job -oozie http://hadoop-master:11000/oozie -config $1 -D nameNode=hdfs://hadoop-master:8020 -D jobTracker=hadoop-master:8032 -D queueName=root -D frequency=60 -D nolockTime=0 -D start=2013-11-22T10:00Z -D end=2014-08-30T00:00Z -run

(注:hadoop2如果是单个hdfs集群,则跟上面差不多,但是如果是多个hdfs集群,就不一样了,其中hdfs://cluster1是core-site.xml中的defaultFs名称,而且没有端口号,jobtracker端口是8032,需要按照如下格式写:)

oozie job -oozie http://hadoop-kf105.jd.com:11000/oozie -config $1 -D nameNode=hdfs://cluster1 -D jobTracker=hadoop-kf100.jd.com:8032 -D frequency=60 -D nolockTime=0 -D start=2013-11-22T10:00Z -D end=2014-08-30T00:00Z -run

执行一个工作流的话可以这样:sh run_oozie.sh oozieTest/job.properties

kill_oozie.sh:

oozie job -oozie http://hadoop-master:11000/oozie -kill $1 

kill一个工作流使用:sh kill_oozie.sh jobId

好了,oozie先介绍到这包,以后有什么新内容再继续补充。

转载于:https://www.cnblogs.com/zhli/p/4823354.html

你可能感兴趣的文章
php面向对象编程(oop)基础知识示例解释
查看>>
关于根据Build Platform或者OS 加载x86或者x64 dll的问题
查看>>
程序员高效开发的几个技巧
查看>>
hexo 搭建博客
查看>>
建造者模式(屌丝专用)
查看>>
Nginx + Tomcat 反向代理 如何在高效的在一台服务器部署多个站点
查看>>
酷狗的皮肤文件存放在哪
查看>>
C++的引用
查看>>
T-SQL查询进阶--深入浅出视图
查看>>
Android读取url图片保存及文件读取
查看>>
完整ASP.Net Excel导入
查看>>
循环队列的运用---求K阶斐波那契序列
查看>>
css3 标签 background-size
查看>>
python itertools
查看>>
Linux内核调试技术——jprobe使用与实现
查看>>
http://lorempixel.com/ 可以快速产生假图
查看>>
编写一个函数isMerge,判断一个字符串str是否可以由其他两个字符串part1和part2“组合”而成...
查看>>
Android Studio默认快捷键
查看>>
函数式编程与参数
查看>>
HDU 2191 【多重背包】
查看>>