Packaging External libs for AWS Glue

tl;dr

PreRequisite

  • setup.py
  • requirements.txt

Python packaging

python setup.py bdist_egg

Pyspark packaging

VERSION=$(shell grep "VERSION = [0-9.]*" setup.py | cut -d\" -f2)
PROJECT_NAME=$(shell grep "PACKAGE_NAME = *" setup.py | cut -d\" -f2)
mkdir -p libs dist
pip install -r requirements.txt -t libs
(cd libs; zip -r ../dist/${PROJECT_NAME}-${VERSION}.zip *)
rm -rf libs/

--

--

--

Perfecting in my job, innovating is my passion

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Moving from Java 8 & CMS to Java 15 & ZGC

Maximum active thread counts

Spring Native: What, Why and How?

Xamarin Forms: Seriously

Integer Programming for Graph Theory and Others with Python: 02 — Shortest Path / BigN

An Effective Way To Share Data Between Controllers In JavaFX

Hack The Box Magic

Create a Custom Connector for Consumption Logic Apps using the Azure Portal

How we scaled our Mysql to handle 3x customers

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Bhavin Tandel

Bhavin Tandel

Perfecting in my job, innovating is my passion

More from Medium

Load data into Redshift from S3

An Introduction to Docker

Programmatically Ingesting Data via Amazon Athena

Building an Apache Airflow configured with Local Executor and Spark Standalone Cluster with Docker