Big Data Developer

Location: Northern Suburbs of Chicago, IL
Date Posted: 06-20-2018

Big Data Developer

What is a Big Data Engineer?

Big data engineers are mainly tasked with transforming data into a format that can be easily analyzed and usable while ensuring an organization's technological infrastructure operates seamlessly in support of business goals.
They do this by:
  • Developing, maintaining, and testing infrastructures for data generation
  • Specializing in creating, testing, implementing, and monitoring applications designed to meet an organization's strategic goals
  • Working closely with data scientists
  • Architecting solutions for data scientists that enable them to do their jobs effectively
  • Employing strong analytical skills in evaluating business needs and assisting in the development of the organization's overall strategy.
  • Effectively collaborating with business analysts and project managers on project planning and project reviews.
  • Utilizing excellent technical skills, which can include distributed systems applications like Hive or Kafka, SQL, database technologies like Oracle or DB2, and shell scripting.
  • Creating and implementing applications that meet an organization's technological needs.
A Big Data Engineer’s primary job duties may involve:
  • Writing code for important business components
  • Conducting technical training sessions
  • Serving as a mentor and resource for junior personnel
  • Overseeing the technical aspects of development projects or serve as a team leader for specific projects
  • Explaining business segment services to executive management
  • Ensuring that their work, including the design, testing, and execution, complies with any applicable guidelines
  • Investigating and identifying alternative solutions to ensure business needs are always met
  • Utilizing their technical skills and critical thinking skills in order to select appropriate methods and evaluation criteria for getting results
About this Opportunity
Our client is looking to hire a software engineer with either Java/Apache Big Data skills.
The role requires working closely with others, frequently in a matrixed environment, and with little supervision. The role requires a self-starter who is proficient in problem solving and requires staying well-informed of technological advancements and putting new innovations into effective practice.  You’ll also be responsible for the development of high-performance, distributed computing tasks using Big Data technologies such as Hadoop, NoSQL and text mining.
  • Bachelor’s degree in Computer Science or related field of study or work in a Big Data / Text Analytics environment (Relevant work experience in creating distributed computing systems applications will be beneficial)
  • 2+ years’ experience working with big data as a data analyst in a large corporate environment
  • Understanding of best practices and standards for Hadoop Distributed File System (HDFS)
  • Any of the following components (at least three):
    • Hadoop HDFS
    • Languages:
      Experience in REST API development
      Knowledge of standard web technologies (HTTP, HTTPS, HTML5) 
Object Oriented Programming using Java
  • Experience with data serialization: JSON or Avro
  • Experience with at least one of NoSQL: Cassandra, MangoDB or Hive or HBase (Graph Models & Spark are both nice to have)
  • Experience with Messaging: Kafka, Flume or Storm
  • Experienced and comfortable with Unstructured Data extract, preparation and processing
  • Basic data modeling experience using Big Data Technologies
  • Experience working in a Unix environment
  • A quick learner with a willingness to do your own research for answers to questions and comfortable attempting solutions on your own. 

  • Responsible for building and supporting a NoSQL and Hadoop-based ecosystem designed for enterprise-wide analysis of structured, semi-structured, and unstructured data
  • Using Big Data programming languages and technology, writing code, completing programming and documentation, and testing and debugging of applications
  • Analyze, design, program, debug and modify software enhancements and/or new products used in distributed, large scale analytics and visualization solutions
  • Interacting with data scientists and industry experts to understand how data needs to be converted, loaded and presented
  • Working with Hadoop/Spark clusters
  • Support regular requests to move data from one cluster to another
  • Bring new data sources into HDFS, transform and load to databases
  • Work collaboratively with Data Scientists and business and IT leaders throughout the company to understand Big Data needs and use cases.
  • Utilizing Machine Learning frameworks for the next generation of applications and platforms using the latest Big Data technologies for large scale enterprise application
  • Develops and maintains system documentation for new and existing applications
  • Develop RESTful APIs and has knowledge of developing and leveraging use Web Services 
  • Collaborate with cross-functional teams - project management, architects, data science team member etc. - to produce the best solutions possible 
  • Strive for continuous improvement of code quality and development practices 
  • Deliver results through collaboration 
  • Translate functional and technical requirements into detailed design 
  • Scaling up machine learning models that are creating business value into highly automated products that act as supply chain decision support systems
  • Following best-in-class software development practices such as agile work flow management and leveraging platforms such as Jira to ensure quality and timeliness of product delivery
  • Identifying creative ideas for integration of machine learning based solutions into business processes to optimize business performance metrics

Attitude and Aptitude for Learning

Paramount to their consideration will be an enthusiasm to learn and a wiliness to be a hands-on self-starter.
this job portal is powered by CATS