Data Engineer (AWS cloud)1006697
At Idego Group, you’ll work with people who find pleasure in programming and have deep knowledge about a variety of technologies. You’ll work for our clients worldwide and provide support in great software development, including IoT, machine learning and blockchain related projects.
We provide quality and give ourselves a lot of autonomy, common sense and general friendliness.
Our client, international provider of e-commerce solutions is expanding the Data Engineering team and we’re helping!
There’s a lot to be done within a pretty nice stack: everything is in the cloud, there are NO on premises solutions.
Main challenges that await you are: migration to a new warehouse which is 100% cloud based and wider implementation of data streaming.
You’ll join two experienced Data Engineers and cooperate with Europe -based Data Science teams.
- Data Lake (AWS S3, EMR, Glue, Athena, Kinesis, Apache Hudi, Delta Lake)
- Data Warehouse (AWS Redshift)
- ETL (Airflow, Python, Spark)
- Infrastructure (Kubernetes, Docker, CI/CD)
- Python (pandas, numpy, boto3)
- 3rd party APIs (Salesforce, Marketo, Zendesk, Zuora)
What we do:
- Building the Data Lake in AWS where we process 500M+ records daily
- Maintaining Data Warehouse (AWS Redshift)
- Constructing ETL data pipelines in Airflow, Python and Spark
- Standardizing and cleaning the data from many different sources (SQL, NoSQL DBs, APIs, queues) for internal reporting
- Architecting scalable Data Streaming is our next big milestone
What expertise we are looking for?
- Data Streaming (Kafka, Kinesis, Spark)
- ETL pipelines in Python
- Docker and Kubernetes
- Advanced SQL and performance tuning
- Git version control system
What perks are waiting for you:
- Work environment with zero micromanagement – we cherish autonomy.
- 100% remote work (unless you want to work from our HQ Gdynia), recruitment & onboarding.
- Really cool seaside apartment available for free for both leisure & work.
- Experienced team from 8 to 15+ years in commercial projects.
- Unique memes channel.
- Private medical insurance and Multisport.
We are neither the agency giving you projects from time to time, nor huge corporation where you are a “dev XYZ”. At Idego – you matter!