Description
The Big Data Engineer will design, develop, and maintain scalable Big Data solutions using Hadoop and Spark technologies. Responsibilities include building and optimizing ETL and data processing pipelines, ingesting data from multiple sources, analyzing structured and unstructured data, optimizing Spark jobs, working with data warehouses and data lakes, collaborating with data stakeholders, troubleshooting production issues, and implementing data quality, governance, and security practices. The role also supports cloud-based data platforms and migration initiatives. Required qualifications include seven years of Big Data Engineering experience, Python and PySpark expertise, Apache Spark and Hadoop ecosystem knowledge, Sqoop, SQL, HDFS, Hive, Yarn, Spark SQL, data modeling, ETL, Linux, shell scripting, Git, and CI/CD experience. A bachelor's or master's degree in a relevant field is required, and cloud, Kafka, Airflow, Databricks, Snowflake, and Agile/Scrum experience are preferred.
