Data Engineer

About Us

Job Description

Key Responsibilities

Data Preparation

• Perform data cleansing, curation, and normalization to ensure data accuracy, consistency, and readiness for analysis and reporting.
• Standardize data formats, units, and structures across datasets to enable seamless integration and usability.

Data Pipeline Development

• Design, develop, and maintain efficient data pipelines for extracting, transforming, and loading (ETL/ELT) data from various sources into data warehouses or data lakes.
• Ensure pipelines are scalable, maintainable, and optimized for performance.

Data Integration

• Collaborate with stakeholders to understand business requirements and integrate data from diverse sources such as APIs, databases, and flat files.
• Work with structured and unstructured data formats, including JSON, XML, CSV, and others.

Data Storage and Management

• Design and optimize relational and non-relational database schemas to support analytics and reporting requirements.
• Maintain data storage systems to ensure data integrity, security, and accessibility.

Automation and Scripting

• Develop Python and SQL scripts for data validation, monitoring, and error handling in pipelines.

Data Quality and Monitoring

• Implement data validation and quality checks to ensure the accuracy and consistency of data.
• Develop monitoring solutions to proactively detect and resolve pipeline failures or anomalies.

Collaboration and Documentation

• Work closely with data analysts, data scientists, and business teams to provide reliable datasets and insights.
• Document data processes, workflows, and architectural decisions for team reference and compliance.

Required Skills and Experience

Essential Skills

• Proficiency in SQL (3+ years) for querying and managing relational databases.
• Strong Python programming skills (2+ years), with experience in libraries likepandas, numpy, pyodbc, sqlalchemy, and requests.
• Experienced with ETL/ELT processes and tools (e.g., Azure Data Factory, Apache Airflow, SSIS).
• Knowledge of data warehousing concepts and platforms (e.g., Azure Synapse, Snowflake, Redshift, BigQuery).
• Understanding of cloud data ecosystems (e.g., Azure).

Preferred Skills

• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Knowledge of data visualization and BI tools (e.g., Power BI, Tableau).
• Exposure to data governance, privacy, and compliance frameworks.
• Familiarity with big data tools and frameworks (e.g., Hadoop, Spark).

General

• Proven experience in large enterprise projects.
• Experience in working following Agile principles and ceremonies.
• Experience in working using issue tracking systems (like jira, clickup).

Personal Attributes

• Strong problem-solving and analytical skills.
• Effective communication and ability to work collaboratively in a team.
• Self-motivated with a willingness to learn and adapt to new tools and technologies.
• Ability to work efficiently under pressure and meet tight deadlines.
• Understanding of Corporate Culture.

What We Offer

• Competitive salary and benefits package.
• Opportunity to work with cutting-edge AI technology in the food industry.
• Collaborative and innovative work environment.
• Career growth and development opportunities.