Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update cpueff spark #278

Merged
merged 3 commits into from
Aug 20, 2024
Merged

Update cpueff spark #278

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7b0c51b
Update cpueff common_utils
nikodemas Jul 10, 2024
bba8fa3
Increase spark memory
nikodemas Jul 16, 2024
a4ebc48
Merge tag 'cpueff-0.0.28'
nikodemas Aug 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cpueff-goweb/spark/cron4cpueff_goweb.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ function run_spark() {
--conf spark.shuffle.useOldFetchProtocol=true
--conf "spark.driver.bindAddress=0.0.0.0" --conf "spark.driver.host=${K8SHOST}"
--conf "spark.driver.port=${PORT1}" --conf "spark.driver.blockManager.port=${PORT2}"
--driver-memory=8g --executor-memory=8g --packages org.apache.spark:spark-avro_2.12:3.4.0
--driver-memory=16g --executor-memory=16g --packages org.apache.spark:spark-avro_2.12:3.4.0
)

py_input_args=(--hdfs_out_dir "$hdfs_out_dir" --start_date "$start_date" --end_date "$end_date")
Expand Down
9 changes: 2 additions & 7 deletions cpueff-goweb/spark/utils.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,13 +158,8 @@ function util_kerberos_auth_with_keytab() {
# setup hadoop and spark in k8s
#######################################
function util_setup_spark_k8s() {
# check hava home
util_set_java_home

hadoop-set-default-conf.sh analytix 'hadoop spark' 3.2
source hadoop-setconf.sh analytix 3.2 spark3
export SPARK_LOCAL_IP=127.0.0.1
export PYSPARK_PYTHON=/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc8-opt/bin/python3
hadoop-set-default-conf.sh analytix
source hadoop-setconf.sh analytix 3.3 spark3
# until IT changes this setting, we need to turn off info logs in this way. Don't try spark.sparkContext.setLogLevel('WARN'), doesn't work, since they are not spark logs but spark-submit logs.
sed -i 's/rootLogger.level = info/rootLogger.level = warn/g' "$SPARK_CONF_DIR"/log4j2.properties
}
Expand Down
Loading