Data Engineer (AWS cloud)

At Idego Group, you’ll work with people who find pleasure in programming and have deep knowledge about a variety of technologies. You’ll work for our clients worldwide and provide support in great software development, including IoT, machine learning and blockchain related projects.

We provide quality and give ourselves a lot of autonomy, common sense and general friendliness.

Project specifics

Our client, international provider of e-commerce solutions is expanding the Data Engineering team and we’re helping!

There’s a lot to be done within a pretty nice stack: everything is in the cloud, there are NO on premises solutions.
Main challenges that await you are: migration to a new warehouse which is 100% cloud based and wider implementation of data streaming.
You’ll join two experienced Data Engineers and cooperate with Europe -based Data Science teams.

Technology stack:

• Data Lake (AWS S3, EMR, Glue, Athena, Kinesis, Apache Hudi, Delta Lake)
• Data Warehouse (AWS Redshift)
• ETL (Airflow, Python, Spark)
• Infrastructure (Kubernetes, Docker, CI/CD)
• Python (pandas, numpy, boto3)
• 3rd party APIs (Salesforce, Marketo, Zendesk, Zuora)

What we do:

• Building the Data Lake in AWS where we process 500M+ records daily
• Maintaining Data Warehouse (AWS Redshift)
• Constructing ETL data pipelines in Airflow, Python and Spark   
• Standardizing and cleaning the data from many different sources (SQL, NoSQL DBs, APIs, queues) for internal reporting
• Architecting scalable Data Streaming is our next big milestone

[culture-book]

What expertise we are looking for?

• Data Streaming (Kafka, Kinesis, Spark) 
• ETL pipelines in Python
• Docker and Kubernetes 
• Advanced SQL and performance tuning
• Git version control system

What perks are waiting for you:

• Work environment with zero micromanagement – we cherish autonomy.
• 100% remote work (unless you want to work from our HQ Gdynia), recruitment & onboarding.
• Really cool seaside apartment available for free for both leisure & work.
• Experienced team from 8 to 15+ years in commercial projects.
• Unique memes channel.
• Private medical insurance and Multisport.
  
  We are neither the agency giving you projects from time to time, nor huge corporation where you are a “dev XYZ”. At Idego – you matter!