Hudi 0.14.0编译
1 编译环境
| Java | 1.8 |
|---|---|
| maven | 3.9.3 |
| hadoop | 3.3.4 |
| hive | 3.1.3 |
| spark | 3.2.1 |
| flink | 1.16.0 |
| hudi | 0.14.0 |
2 hudi准备
2.1 源码
$ git clone https://github.com/apache/hudi.git
$ cd hudi
$ git checkout release-0.14.0
2.2 修改pom文件
2.2.1 新增repository加速依赖下载
<repository>
<id>nexus-aliyun</id>
<name>nexus-aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
2.2.2 修改依赖组件版本
hudi/pom.xml
更改spark版本为3.2.3，flink版本为1.16.0
<spark32.version>3.2.3</spark32.version>
<flink1.16.version>1.16.0</flink1.16.version>
hudi/packaging/hudi-flink-bundle/pom.xml
更改hive版本为3.1.3
参考链接：Hive Metastore | Apache Hudi
<profile>
<id>flink-bundle-shade-hive3</id>
<properties>
<hive.version>3.1.3</hive.version>
<flink.bundle.hive.scope>compile</flink.bundle.hive.scope>
</properties>
<dependencies>
<dependency>
<groupId>${hive.groupid}</groupId>
<artifactId>hive-service-rpc</artifactId>
<version>${hive.version}</version>
<scope>${flink.bundle.hive.scope}</scope>
</dependency>
<dependency>
<groupId>${hive.groupid}</groupId>
<artifactId>hive-standalone-metastore</artifactId>
<version>${hive.version}</version>
<scope>${flink.bundle.hive.scope}</scope>
</dependency>
</dependencies>
</profile>
3 编译
3.1 编译命令
mvn clean package \
-DskipTests \
-Dcheckstyle.skip \
-Dhadoop.version=3.3.4 \
-Dhive.version=3.1.3 \
-Dscala-2.12 \
-Dspark3.2 \
-Dflink1.16 \
-Pflink-bundle-shade-hive3
3.2 编译成功
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hudi 0.14.0:
[INFO]
[INFO] Hudi ............................................... SUCCESS [ 44.589 s]
[INFO] hudi-tests-common .................................. SUCCESS [01:07 min]
[INFO] hudi-common ........................................ SUCCESS [01:25 min]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 18.216 s]
[INFO] hudi-sync-common ................................... SUCCESS [ 5.951 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [ 26.903 s]
[INFO] hudi-aws ........................................... SUCCESS [ 12.256 s]
[INFO] hudi-timeline-service .............................. SUCCESS [ 5.442 s]
[INFO] hudi-client ........................................ SUCCESS [ 0.056 s]
[INFO] hudi-client-common ................................. SUCCESS [ 19.466 s]
[INFO] hudi-spark-client .................................. SUCCESS [01:04 min]
[INFO] hudi-spark-datasource .............................. SUCCESS [ 0.028 s]
[INFO] hudi-spark-common_2.12 ............................. SUCCESS [01:32 min]
[INFO] hudi-spark3-common ................................. SUCCESS [ 25.230 s]
[INFO] hudi-spark3.2plus-common ........................... SUCCESS [ 21.612 s]
[INFO] hudi-spark3.2.x_2.12 ............................... SUCCESS [ 36.385 s]
[INFO] hudi-java-client ................................... SUCCESS [ 8.972 s]
[INFO] hudi-spark_2.12 .................................... SUCCESS [01:25 min]
[INFO] hudi-gcp ........................................... SUCCESS [ 7.062 s]
[INFO] hudi-utilities_2.12 ................................ SUCCESS [ 32.803 s]
[INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [01:19 min]
[INFO] hudi-cli ........................................... SUCCESS [ 27.285 s]
[INFO] hudi-flink-client .................................. SUCCESS [ 19.600 s]
[INFO] hudi-datahub-sync .................................. SUCCESS [ 3.451 s]
[INFO] hudi-adb-sync ...................................... SUCCESS [ 6.029 s]
[INFO] hudi-sync .......................................... SUCCESS [ 0.021 s]
[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [ 35.741 s]
[INFO] hudi-datahub-sync-bundle ........................... SUCCESS [ 40.215 s]
[INFO] hudi-hive-sync-bundle .............................. SUCCESS [ 32.659 s]
[INFO] hudi-aws-bundle .................................... SUCCESS [ 48.612 s]
[INFO] hudi-gcp-bundle .................................... SUCCESS [ 32.596 s]
[INFO] hudi-spark3.2-bundle_2.12 .......................... SUCCESS [01:00 min]
[INFO] hudi-presto-bundle ................................. SUCCESS [ 41.722 s]
[INFO] hudi-utilities-slim-bundle_2.12 .................... SUCCESS [01:01 min]
[INFO] hudi-timeline-server-bundle ........................ SUCCESS [ 47.789 s]
[INFO] hudi-trino-bundle .................................. SUCCESS [ 37.650 s]
[INFO] hudi-examples ...................................... SUCCESS [ 0.023 s]
[INFO] hudi-examples-common ............................... SUCCESS [ 6.493 s]
[INFO] hudi-examples-spark ................................ SUCCESS [ 17.247 s]
[INFO] hudi-flink-datasource .............................. SUCCESS [ 0.028 s]
[INFO] hudi-flink1.16.x ................................... SUCCESS [ 4.574 s]
[INFO] hudi-flink ......................................... SUCCESS [ 14.562 s]
[INFO] hudi-examples-flink ................................ SUCCESS [ 7.564 s]
[INFO] hudi-examples-java ................................. SUCCESS [ 9.664 s]
[INFO] hudi-flink1.13.x ................................... SUCCESS [ 6.623 s]
[INFO] hudi-flink1.14.x ................................... SUCCESS [ 5.684 s]
[INFO] hudi-flink1.15.x ................................... SUCCESS [ 4.878 s]
[INFO] hudi-flink1.17.x ................................... SUCCESS [ 5.485 s]
[INFO] hudi-kafka-connect ................................. SUCCESS [ 8.019 s]
[INFO] hudi-flink1.16-bundle .............................. SUCCESS [01:54 min]
[INFO] hudi-kafka-connect-bundle .......................... SUCCESS [01:20 min]
[INFO] hudi-cli-bundle_2.12 ............................... SUCCESS [ 44.297 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26:05 min
[INFO] Finished at: 2023-10-24T14:54:46+08:00
[INFO] ------------------------------------------------------------------------
编译成功后，进入hudi-cli说明成功。
[root@localhost hudi]# ./hudi-cli/hudi-cli.sh
编译完成后，相关的包在packaging目录的各个模块中：
[root@localhost packaging]# pwd
/root/app/hudi/packaging
[root@localhost packaging]# ll
总用量 4
drwxr-xr-x. 11 root root 247 10月 24 11:11 bundle-validation
drwxr-xr-x. 4 root root 46 10月 24 14:43 hudi-aws-bundle
drwxr-xr-x. 5 root root 89 10月 24 14:54 hudi-cli-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:42 hudi-datahub-sync-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:50 hudi-flink-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:44 hudi-gcp-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:42 hudi-hadoop-mr-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:43 hudi-hive-sync-bundle
drwxr-xr-x. 3 root root 32 10月 24 13:45 hudi-integ-test-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:52 hudi-kafka-connect-bundle
drwxr-xr-x. 2 root root 21 10月 24 13:45 hudi-metaserver-server-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:46 hudi-presto-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:45 hudi-spark-bundle
drwxr-xr-x. 4 root root 101 10月 24 14:48 hudi-timeline-server-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:48 hudi-trino-bundle
drwxr-xr-x. 4 root root 46 10月 24 14:39 hudi-utilities-bundle
drwxr-xr-x. 4 root root 63 10月 24 14:46 hudi-utilities-slim-bundle
-rw-r--r--. 1 root root 2206 10月 24 11:11 README.md
比如，flink与hudi的包，hudi-flink1.16-bundle-0.14.0.jar：
[root@localhost target]# pwd
/root/app/hudi/packaging/hudi-flink-bundle/target
[root@localhost target]# ll
总用量 135228
drwxr-xr-x. 4 root root 33 10月 24 14:50 classes
-rw-r--r--. 1 root root 13474 10月 24 14:52 dependency-reduced-pom.xml
drwxr-xr-x. 3 root root 25 10月 24 14:50 generated-sources
-rw-r--r--. 1 root root 96299310 10月 24 14:52 hudi-flink1.16-bundle-0.14.0.jar
-rw-r--r--. 1 root root 42111892 10月 24 14:52 hudi-flink1.16-bundle-0.14.0-sources.jar
drwxr-xr-x. 2 root root 28 10月 24 14:50 maven-archiver
drwxr-xr-x. 3 root root 22 10月 24 14:50 maven-shared-archive-resources
drwxr-xr-x. 3 root root 35 10月 24 14:50 maven-status
-rw-r--r--. 1 root root 18561 10月 24 14:50 original-hudi-flink1.16-bundle-0.14.0.jar
-rw-r--r--. 1 root root 14046 10月 24 14:50 original-hudi-flink1.16-bundle-0.14.0-sources.jar
-rw-r--r--. 1 root root 809 10月 24 14:50 rat.txt
drwxr-xr-x. 3 root root 22 10月 24 14:50 test-classes
评论