Big Data Engineer Chennai | Pune
Experience Range: 6+ Years
Location: Chennai/Pune (Hybrid – 2 Days Work From Office in a Week)
Job description:
Job Summary:
We are looking for a core team member to join our Agile development team, where the primary responsibility will be to build backend data applications and services that support our Analytics platforms. The role will require hands-on expertise in technologies such as SCALA, SPARK, PYTHON, and SHELL SCRIPTING, as well as a basic understanding of AWS cloud services for ongoing cloud migration efforts. The ideal candidate will collaborate closely with product owners and team members to define, implement, and optimize solutions that address various challenges and demands related to analytics and data processing.
Responsibilities:
- Agile Team Collaboration and Development:
As a core member of our Agile team, you will contribute to the design and development of backend data applications. These applications and services will be the foundation for our Analytics platforms, ensuring that the data processing systems are reliable, scalable, and optimized to support high volumes of data for analytical purposes. - Technical Expertise in SCALA, SPARK, PYTHON, and SHELL SCRIPTING:
A key part of your role will be applying your hands-on expertise in SCALA, SPARK, PYTHON, and SHELL SCRIPTING to develop and maintain data processing pipelines, create efficient data models, and automate tasks. You will be tasked with building robust back-end solutions and implementing best practices for the technologies mentioned. Your experience will enable you to address complex data challenges, process large datasets, and optimize data flows for faster, more efficient processing. - AWS Cloud Migration Knowledge:
While the primary focus will be on backend development, having basic knowledge of AWS will be crucial as part of the ongoing cloud migration effort. You will be expected to help migrate legacy systems to the cloud, ensuring that data infrastructure is cloud-native, scalable, and secure. Familiarity with AWS services like EC2, S3, and Lambda will be beneficial for this aspect of the role. - Collaboration with Product Owners and Team Members:
Working alongside product owners, business analysts, and other development team members, you will contribute to defining project requirements and developing implementation strategies. This will involve iterative development cycles and regular stand-ups to ensure all deliverables are aligned with business goals and timelines. - Troubleshooting and Optimization:
You will partner with business stakeholders and other technical resources to troubleshoot data issues and ensure system performance. This includes managing data capacity, identifying bottlenecks, and optimizing processing workflows to ensure the infrastructure supports growing demands and future scalability. - Automation and Scheduling:
Automation will be a key part of the role, and you will be responsible for creating scripts and processes to automate data processing, ETL pipelines, and related workflows. Experience with AUTOsys, a popular scheduling tool, is a major advantage as it will assist in automating batch jobs and scheduling data-related tasks. Experience with Jenkins for continuous integration and deployment (CI/CD) pipelines is also a significant plus, as it allows for streamlined development and deployment processes.
Required Skills and Experience:
Communication: Strong verbal and written communication skills, as you will work with diverse teams and stakeholders to ensure clear understanding of requirements and progress.
Pan Asia HR Solution aims to revolutionize the workforce landscape. With a customer-centric approach, we deliver bottom-line benefits and work effectively with our clients. Our extensive reach, market expertise, and innovative business models ensure that we stay ahead of the curve.
Expertise in SCALA, SPARK, PYTHON, SHELL SCRIPTING: Proven ability to work hands-on with these programming languages, developing complex backend systems and data processing workflows.
AWS Knowledge: Basic experience with AWS, particularly in relation to cloud migration efforts and cloud-based infrastructure.
Jenkins Pipeline: Experience with Jenkins and continuous integration/deployment pipelines is highly desired, enabling efficient, automated software development processes.
AUTOsys: Familiarity with AUTOsys or similar job scheduling tools is a plus, as the tool is key to automating and managing workflows and data processes.
Agile Development: Ability to work within an Agile development team, collaborating closely with other developers and stakeholders to deliver high-quality solutions.
Data Analysis & Troubleshooting: Strong problem-solving skills to troubleshoot data issues, optimize data pipelines, and identify areas for improvement in performance and scalability.
Pan Asia HR Solution is a progressive and innovative IT Workforce Solutions company. It was incorporated with a mission to serve Indian Customers and Global customers in India.
Pan Asia does not charge candidates any fee at any stage of the recruitment process
