To switch the python version in pyspark, set the following environment variables. I was working in an environment with Python2 and Python3. I had to use Python3 in pyspark where the spark was using Python 2 by default.
Python 2 was pointing to –> /usr/bin/python
Python 3 was pointing to –> /usr/bin/python3
To configure pyspark to use python 3, set the following environment variables.
export PYSPARK_PYTHON=/usr/bin/python3 export PYSPARK_DRIVER_PYTHON=/usr/bin/python3
Now type pyspark in the commandline.