Data Quality Developer - Java and Spark LT Contract
· Hands-on expertise with ETL pipelines using internal framework written in Apache Spark & Java
· Acquire data from primary or secondary data sources and maintain databases/data systems
· Identify, analyze, and interpret trends or patterns in complex data sets
· Manage data ingestion in real time Kafka/REST APIs for streaming data from Graph using Spark
· Develop and implement databases, data collection systems, data analytics and other strategies that optimize data efficiency and quality
· Filter and clean data by reviewing reports and performance indicators to locate and correct problems
Required Technical Skills:
· Strong hands-on development experience with both Spark, Kafka
· Strong Java experience particularly within the data space
· Experience in Big Data related technologies like AWS, Hadoop, HIVE, Presto
· Experience implementing complex ETL pipelines with Spark toolset.
· Experience in Graph
· Experience working in an Agile Team environment.
· Kafka, NoSQL, Scala, MongoDB
· Experience with large data volumes, processing, transforming and transporting large-scale data