To switch the python version in pyspark, set the following environment variables. I was working in an environment with Python2 and Python3. I had to use Python3 in pyspark where the spark was using Python 2 by default.

Python 2 was pointing to –> /usr/bin/python

Python 3 was pointing to –> /usr/bin/python3

To configure pyspark to use python 3, set the following environment variables.

export PYSPARK_PYTHON=/usr/bin/python3
export PYSPARK_DRIVER_PYTHON=/usr/bin/python3

Now type pyspark in the commandline.