Originally published in The New Stack Update.
Stitch Data analyzed everyone on LinkedIn that said they are a data engineers. They found that many were originally a developer, with 42 percent being a software engineer in their last job. This makes sense, as big data training specialist Jesse Anderson says a data engineer’s primary language should be Java, with additional skills needed in SQL and a dynamic language like Python or Scala. While they are not database administrators (DBAs), they are more likely to know about business intelligence (BI) and extract, transform, load (ETL) as compared to other developers. Yet, the biggest difference between data engineers and software engineers is that the latter is 11 percentage points more likely to use Hadoop. The next time you read about a shortage of data engineers you’ll know why you need to learn Hadoop as opposed to the more dynamically growing Spark.
