Senior Data Engineer New York, NY - Hybrid position Onsite 2 days a week (local to area) Rate $100-$110 per hour 1+ year contract Must be a USC to obtain L2 security clearance No Corp to Corp or 3rd party agencies We are looking for a Senior Data Engineer to join our team of professionals. The selected individual should be well versed in Databricks and AWS. In this position you will: Work on a team of 15 and have expert level skills in coding and testing pipelines in Databricks on AWS 6 Other solid skills Python, SQL, Spark, AWS, Trino Position Description
- Design, develop, monitor, test and maintain data pipelines in Databricks on an AWS 6 ecosystem with Delta Lake, Python, SQL and Starburst as the technology stack. Collaborate with cross-functional teams to understand data needs and translate them into effective data pipeline solutions.
- Establish data quality checks and ensure data integrity and accuracy throughout the data lifecycle.
- Automate testing of the data pipelines and configure as part of CICD
- Optimize data processing and query performance for large-scale datasets within AWS and Databricks environments.
- Document data engineering processes, architecture, and configurations.
- Troubleshooting and debugging data-related issues on the AWS Databricks platform.
- Integrating Databricks with other AWS products such as SNS, SQS, and MSK.
Qualifications
- Minimum of 5 years of experience in data engineering roles, with a focus on AWS and Expert level skills coding and testing Databricks in an AWS 6 Environment.
- Experience with Python, PySpark, SQL, Spark
- Gitlab with CI/CD.
- AWS Services like S3, RDS, Lambda, SQS, SNS, MSK is required
- Databricks Certified is highly desired
- Strong SQL skills to perform data analysis and understanding of source data.
- Experience with data pipeline orchestration tools
- Communication, Collaboration, experimentation, inquisitiveness, detailed oriented
Must be Databricks Certified