spark使运气行

运行环境

1. 在这里插入代码片
2. [root@hadoop000 conf]# java -version
3. java version "1.8.0_144"
4. Java(TM) SE Runtime Environment (build 1.8.0_144-b01)

复制代码

1. [root@hadoop000 conf]# echo $JAVA_HOME
2. /home/hadoop/app/jdk1.8.0_144

复制代码

1. [root@hadoop000 conf]# vi spark-env.sh
2. [root@hadoop000 conf]# mv slaves.template slaves
3. [root@hadoop000 conf]# vi slave

复制代码

步骤

1. /home/hadoop/app/spark-2.2.0-bin-2.6.0-cdh5.7.0/bin/spark-submit \
2. --class org.apache.spark.examples.SparkPi \
3. --master spark://192.168.2.111:7077 \
4. --executor-memory 1G \
5. --total-executor-cores 2 \
6. /home/hadoop/app/spark-2.2.0-bin-2.6.0-cdh5.7.0/examples/jars/spark-examples_2.11-2.2.0.jar

复制代码

spark-shell

1. [root@hadoop000 bin]# /home/hadoop/app/spark-2.2.0-bin-2.6.0-cdh5.7.0/bin/spark-shell \
2. > --master spark://192.168.2.111:7077 \
3. > --executor-memory 2G \
4. > --total-executor-cores 2
5. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
6. Setting default log level to "WARN".
7. To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
8. 25/02/15 16:45:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
9. 25/02/15 16:45:43 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
10. Spark context Web UI available at http://192.168.2.111:4040
11. Spark context available as 'sc' (master = spark://192.168.2.111:7077, app id = app-20250215164538-0002).
12. Spark session available as 'spark'.
13. Welcome to
14.       ____              __
15.      / __/__  ___ _____/ /__
16.     _\ \/ _ \/ _ `/ __/  '_/
17.    /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
18.       /_/
19.          
20. Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
21. Type in expressions to have them evaluated.
22. Type :help for more information.

复制代码

1. scala> sc.textFile("hdfs://192.168.2.102:9000//user/spark/input/word.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).saveAsTextFile("hdfs://192.168.2.102:9000//user/spark/out")

复制代码

1. scala> sc.textFile("hdfs://192.168.2.102:9000/user/spark/out/*").collect().foreach(println)
2. (orange,1)
3. (queen,1)
4. (rabbit,1)
5. (fish,1)
6. (dog,1)
7. (apple,1)
8. (pig,1)
9. (umbrella,1)
10. (snake,1)
11. (lion,1)
12. (juice,1)
13. (cat,1)
14. (tiger,1)
15. (banana,1)
16. (monkey,1)
17. (nose,1)
18. (kite,1)
19. (elephant,1)
20. (ice,1)
21. (goat,1)
22. (horse,1)

复制代码

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。