Lead Data Engineer Full-time/Permanent Onsite - NYCResponsibilities:
- Take ownership of the entire data engineering lifecycle, including design, implementation, performance tuning, and monitoring
- Manage and work with the firm's database managed service provider (MSP) to manage and monitor the firm's evolving database requirements
- Design, develop, and maintain scalable data pipelines and ETL processes critical to support business operations and analytics. (Python / SQL)
- Optimize and tune data systems for high performance and reliability
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions
- Implement and manage data storage solutions in cloud environments (AWS, Azure, Google Cloud)
- Ensure data quality and integrity through rigorous testing and validation
- Continuously evaluate and integrate new technologies and tools to improve data processing capabilities
- Manage distributed scheduled tasks on individual servers to a centralized ruleset
- Work with the development team and database MSP to modernize the backup and recovery strategy for 3-4 SQL servers across different cloud environments
- Support the addition of new tenants on the firm's production data platform
Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field from a top-tier university.
- 4-5 years of relevant professional experience in data engineering
- Strong programming skills in SQL and Python.
- Proficiency in database administration
- Proven experience migrating on-prem data and associated processes to cloud platforms (AWS, Azure, Google Cloud)
- Demonstrated ability to design and implement efficient and reliable data pipelines and ETL processes
- Strong problem-solving skills and ability to troubleshoot complex data issues
- Excellent communication skills and ability to work collaboratively in a team environment
- A proactive mindset with a willingness to take ownership and drive projects to completion. Strong organizational skills along with a desire to continually be challenged
- Experience with monitoring, CI/CD, Load Balancing, and Testing tools (e.g. Redgate, Datadog)