Senior Data Engineer
Paris
Contract Opportunity
 
The candidate will be responsible for optimizing data ingestion using pyspark and adhering to best practices (caching, persist, partitioning, schema inference, etc.). The role also involves maintaining and/or establishing data quality policies, developing in a modular approach, and encapsulating environments. Knowledge of Checkmarx and SonarQube is considered a plus for ensuring code quality.
 
Key Responsibilities:

• Experience using GitHub Actions with multi-branches, including triggers, if conditions, workflows, and jobs
• Experience with the following AWS technologies: EC2, S3, LAMBDA, STEP FUNCTIONS, CLOUDWATCH
• Utilize S3 for storing data at various stages of the ETL process and establish retention policies
• Monitor systems using CLOUDWATCH, with priority given to candidates with experience in GLUE
• Proficiency in Python and PySpark
• Develop features, such as enriching functions or monitoring data, and set up new logs
• Perform technical validation tests, including unit tests for data validation and integration testing
• Deliver technical documentation (schemas, types, tables, and table evolution processes)
• Verify and validate new code versions (expertise in Git, with familiarity with Git branch usage in Databricks is a plus) Proficiency in Git commands is required for orchestrating ingestion pipelines.

Experience Required:

• A minimum of 6 years in data engineering