Job Description
• Professional experience with a cloud platform. Developers must have sound knowledge of Apache Spark and Python programming. Deep experience in developing data processing tasks using pySpark such as reading data from external sources, merging data, performing data enrichment, and loading into target data destinations.
Desired Candidate Profile
- Ability to design, build and unit test the application in Spark/Pyspark.
- In-depth knowledge of Hadoop, Spark, and similar frameworks.
- Ability to understand existing ETL & logic to convert into Spark/PySpark/ Spark SQL.
- Strong hands-on expertise in diverse DevSecOps concepts / tools, especially on Azure DevOps, Pipelines, GitHub, GitHub actions
- Knowledge of Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
- Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
