Aetion
New York City, NYDescription Your primary responsibility will be developing transformation logic against disparate datasets into Aetion's proprietary format.
You will work closely with our Science team in developing this logic, which is in SQL and Scala UDF and executed over a Spark cluster.
In addition, you will be integral in developing and enhancing our ELT process which utilizes Apache Airflow (python), Spark/Databricks and a combination of Java/Scala to develop data ingestion infrastructure.ResponsibilitiesThe following duties include, but are not limited to:Develop transformation logic to convert disparate datasets into Aetion’s proprietary format. Work with the Science team to develop this logic in SQL executed over a Spark cluster.Assess, develop, troubleshoot and enhance ELT process, which utilizes Apache Airflow (python) and a combination of Java/Scala. Modify JavaScript Object Notation (JSON) files to describe the schemas of the datasets, ensuring system functionality through routine maintenance and testing. Work on a full-stack rapid-cycle analytic application. Develop highly effective, performant, and scalable components capable of handling large amounts of data for over 10 million patients. Work with the Science and Product teams to understand and assess client needs, and to ensure optimal system efficiency. QualificationsBachelor’s degree or equivalent in Computer Science, Computer Engineering, Information Systems, or a related field.4 years of experience in the position offered or related position, including 5 years of experience with: designing, developing, maintaining large-scale data ETL pipelines using Java/Scala in AWS, Hadoop, Spark, and DataBricks to manage Apache Spark infrastructure.Experience building backend modules, low latency REST API in distributed environment using Java, Docker, SQL, MVN, Spring, Jenkins.Experience writing complex SQL queries, UDFs to process large amounts of data across relational, non-relational databases, JSON and Spark SQLs.Experience translating requirements from product, DevOps teams to technology solutions using SDLC. Aetion is an Equal Opportunity Employer.
Aetion is committed to being an employer of choice, not just a good place to work, but a great and inclusive place to work.
To that end, we strive to recruit and maintain a workforce that meaningfully represents the diverse and culturally rich communities that we serve.
Qualified applicants will receive consideration for employment without regard to their race, color, religion, national origin, sex, sexual orientation, gender identity, protected veteran status or disabled status or, genetic information.
How to Apply
Follow the application procedure at stackoverflow.com for more info.
Related positions:
Download free Serefind app to explore more!
