By Subhash Sreenivasachar, Software Engineer Technical Lead at Epsilon
AWS plays a pivotal role in helping engineers, data-scientists focus on building solutions and problem solving without worrying about the need to setup infrastructure. With Serverless & pay-as-you-go approach for pricing, AWS provides ease of creating services on the fly.
AWS Glue is widely used by Data Engineers to build serverless ETL pipelines. PySpark being one of the common tech-stack used for development. However, despite the availability of services, there are certain challenges that need to be addressed.
Debugging code in AWS environment whether for ETL script (PySpark) or any other service is a challenge.
- Ongoing monitoring of AWS service usage is key to keep the cost factor under control
- AWS does offer Dev Endpoint with all the spark libraries installed, but considering the price, it’s not viable for use for large development teams
- Accessibility of AWS services may be limited for certain users
Solutions for AWS can be developed, tested in local environment without worrying about accessibility or cost factor. Through this article, we are addressing two problems –
- Debugging PySpark code locally without using AWS dev endpoints.
- Interacting with AWS Services locally
Both problems can be solved with use of Docker images.
- First, we do away the need for a server on AWS environment & instead, a docker image running on the machine acts as the environment to execute the code.
AWS provides a sandbox image which can be used for PySpark scripts. Docker image can be setup to execute PySpark Code. https://aws.amazon.com/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/
- With docker machine available to execute the code, there’s a need for a service like S3 to store (read/write) files while building an ETL pipeline.
Interactions with S3 can be replaced with LocalStack which provides an easy-to-use test/mocking framework for developing Cloud applications. It spins up a testing environment on your local machine that provides the same functionality and APIs as the real AWS cloud environment.
So far, the article deals with building an ETL pipeline and use of services available. However, similar approach can be adapted to any use case while working with AWS services like SNS, SQS, CloudFormation, Lambda functions etc.
- Use docker containers as remote interpreter
- Run PySpark session on the containers
- Spin up S3 service locally using LocalStack
- Use PySpark code to read and write from S3 bucket running on LocalStack
Following tools must be installed on your machine
- PyCharm Professional/ VisualStudio Code
- Download or pull docker images (docker pull <image name>)
- Docker containers can be used as remote interpreters in PyCharm professional version.
With Docker installed and images pulled to your local machine, start setting PyCharm with configurations to start the containers.
- Create a docker-compose.yml file
- Use requirements file with packages to be installed
- Setup Python remote interpreter
- Setup Python interpreter using the docker-compose file.
- Select `glue-service` in PyCharm Docker Compose settings.
- Docker-compose file creates and runs the containers for both images
- LocalStack by default runs on port 4566 and S3 service is enabled on it
- Required libraries to be imported
- Add a file to S3 bucket running on LocalStack
http://host.docker.internal:4566 is the S3 running locally inside docker container
- Setup PySpark session to read from S3
- PySpark session connects to S3 via mock credentials provided
- You can read from S3 directly using the PySpark session created
- Finally, it’s possible to write to S3 in any preferred format
Once the above-mentioned steps have been followed, we can create a dummy csv file with mock data for testing and you should be good to
- Add file to S3 (which is running on LocalStack)
- Read from S3
- Write back to S3 as parquet
You should be able to run the .py file to execute & PySpark session will be created that can read from S3 bucket which is running locally using LocalStack API.
Additionally, you can also check if LocalStack is running with http://localhost:4566/health
LocalStack provides you ability to run commands using AWS CLI as well.
Use of Docker & Localstack provides a quick and easy way to run Pyspark code, debug on containers and write to S3 which is running locally. All this without having to connect to any AWS service.
Bio: Subhash Sreenivasachar is Lead Software Engineer at Epsilon Digital Experience team, building engineering solutions to solve data science problems specifically personalization, and help drive ROI for clients.