Apache Spark cluster installation on Kubernetes | Apache Spark Helm Chart
4.9 هزار بار بازدید -
2 سال پیش
-
#spark
#spark #kubernetes #helm
Apache Spark is one of the most widely used open-source clusters for big data processing. Installing Spark on Kubernetes using a helm chart is a great way to quickly deploy and manage a Spark cluster in a production environment. In this video, we will discuss the process of setting up a Spark cluster on Kubernetes using a helm chart and the various options available to customize your cluster.
Please subscribe by clicking the below link to support my channel:
https://bit.ly/3gM6elC
Commands to execute in kubernetes master node.
./helm repo add bitnami-repo https://charts.bitnami.com/bitnami
./helm install spark-release bitnami-repo/spark
If you want to scale up worker nodes please use this command
------------------------------------------------------------------------------------------------------
./helm upgrade spark-release bitnami-repo/spark --set worker.replicaCount=5
To expose the spark master pod use this command
------------------------------------------------------------------------------------
kubectl port-forward --address 0.0.0.0 pod/spark-release-master-0 30010:8080
To access spark master
---------------------------------------
http://10.0.0.10:30010/
spark submit example
-------------------------------------
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://spark-release-master-0.spark-release-headless.default.svc.cluster.local:7077 --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples_2.12-3.3.1.jar 10
YouTube keywords
------------------------------
spark on kubernetes, spark kubernetes, spark, spark kubernetes, spark kubernetes tutorial, spark kubernetes configuration, spark kubernetes helm, spark kubernetes client mode, install spark on kubernetes, spark submit kubernetes, spark bitnami, spark deployment on kubernetes, apache spark kubernetes, spark helm, helm commands, spark submit jar file example, kubernetes, helm chart tutorial
Apache Spark is one of the most widely used open-source clusters for big data processing. Installing Spark on Kubernetes using a helm chart is a great way to quickly deploy and manage a Spark cluster in a production environment. In this video, we will discuss the process of setting up a Spark cluster on Kubernetes using a helm chart and the various options available to customize your cluster.
Please subscribe by clicking the below link to support my channel:
https://bit.ly/3gM6elC
Commands to execute in kubernetes master node.
./helm repo add bitnami-repo https://charts.bitnami.com/bitnami
./helm install spark-release bitnami-repo/spark
If you want to scale up worker nodes please use this command
------------------------------------------------------------------------------------------------------
./helm upgrade spark-release bitnami-repo/spark --set worker.replicaCount=5
To expose the spark master pod use this command
------------------------------------------------------------------------------------
kubectl port-forward --address 0.0.0.0 pod/spark-release-master-0 30010:8080
To access spark master
---------------------------------------
http://10.0.0.10:30010/
spark submit example
-------------------------------------
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://spark-release-master-0.spark-release-headless.default.svc.cluster.local:7077 --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples_2.12-3.3.1.jar 10
YouTube keywords
------------------------------
spark on kubernetes, spark kubernetes, spark, spark kubernetes, spark kubernetes tutorial, spark kubernetes configuration, spark kubernetes helm, spark kubernetes client mode, install spark on kubernetes, spark submit kubernetes, spark bitnami, spark deployment on kubernetes, apache spark kubernetes, spark helm, helm commands, spark submit jar file example, kubernetes, helm chart tutorial
2 سال پیش
در تاریخ 1401/08/17 منتشر شده
است.
4,965
بـار بازدید شده