AmirHossein Sojoodi

AmirHossein Sojoodi

PhD student in Computer Engineering at Queen's University

Setup Apache Spark

Published:
1 minute read

Instruction to setup an Apache Spark cluster - Standalone mode (Ubuntu 16.04)

First make sure you have installed all the dependencies in all nodes. (master and clients):

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
sudo apt-get install scala

Create a user of same name in master and all slaves to make your tasks easier during ssh and also switch to that user in master.

Add entries in hosts file (master and slaves)

Edit hosts file.

sudo vim /etc/hosts

And add entries of master and slaves in hosts file

<MASTER-IP> master
<SLAVE01-IP> slave01
<SLAVE02-IP> slave02

Install Open SSH Server-Client and Generate key pairs, then copy your key to all nodes to make passwordless ssh available.

sudo apt-get install openssh-server openssh-client
ssh-keygen -t rsa -P ""
ssh-copy-id your-user@slave-01
ssh-copy-id your-user@slave-02

Download spark binaries, extract it and put it in (/usr/local/spark)

wget http://www-us.apache.org/dist/spark/spark-2.3.2/spark-2.3.2-bin-hadoop2.7.tgz
tar xvf spark-2.3.2-bin-hadoop2.7.tgz
sudo mv spark-2.3.2-bin-hadoop2.7 /usr/local/spark

Set up the environment for Spark

vim ~/.bashrc
# add these two lines at the end of the file(remove the #s):
# export SPARK_HOME="/usr/loca/spark"
# export PATH=$PATH:/usr/local/spark/bin
source ~/.bashrc

Make a copy of spark-env-template and Update its content

cd /usr/local/spark/conf
cp spark-env.sh.template spark-env.sh
vim spark-env.sh
# add these two lines to the file (remove the #s):
# export SPARK_MASTER_HOST='SERVER-IP'
# export JAVA_HOME=<JAVA_PATH>

Add these lines to the slave file (in the current directory):

master
slave01
slave02

mvn clean install

You May Also Enjoy

Utilize PTX Just-In-Time (JIT) Compilation in CUDA

4 minute read

Published:

In this post I’ve written about how to utilize PTX Just-In-Time (JIT) compilation in CUDA. PTX is a low-level assembly-like language that is used to represent the GPU code. The PTX code is then compiled to the machine code by the NVIDIA driver at runtime. This process is called Just-In-Time (JIT) compilation. But before I write about how to use PTX JIT compilation, I’ll provide some background on why you might want to use it.

Collection of my dotfiles (public version)

1 minute read

Published:

This is a collection of my dotfiles that I use on my Linux system. I have written about them in various posts, but I thought I can put them all in one place. You can checkout the github repository here, too.

Tmux Tips and Tricks

10 minute read

Published:

(If you are already familiar with tmux, and you are looking for a simple .tmux.conf, go to the end of this post.)

Setup LAMMPS

4 minute read

Published:

Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) is a classical molecular dynamics code that can be used to model atoms or, more generally, as a parallel particle simulator at various scales. The complete documentation of LAMMPS can be found here. In this post, I will provide a guide on how to setup LAMMPS on a Linux machine. My setup is on a cluster with NVIDIA GPUs, UCX, and OpenMPI. Also, we have a built-in module system to load the necessary modules.