While we know that data engineers are very different than data architects — as the latter conceptualize data frameworks and the former build and maintain them — the data engineer function has evolved quite a bit in recent years. Particularly, roles and skills have moved toward greater specializations in data security. Here, find out more about data engineers and, specifically, the role they play in maintaining data security.
Five to ten years ago, data engineers could receive certifications for data security, but they were heavily focused on IT considerations. Nowadays, data security and privacy are about 40% technical and 60% a governance problem (and, of course, poor data governance can lead to regulatory and security concerns).
Over time, some market transformation made security considerations a critical aspect of data engineering. For instance:
- The rising interest of hackers on data-driven services (like recommender systems)
- Cloud-first strategies became frequent in data labs and other business departments and, with them, numerous Shadow IT cases left data engineers in charge of deployments in production autonomously
Security governance became part of the daily considerations of data engineers and, in light of the transformations to comply with regulations like the General Data Protection Regulation (GDPR) in the European Union and the California Consumer Privacy Act (CCPA) in the state of California, the role that data engineers plays in data protection and security has come to the forefront. Here’s how data engineers are involved in making sure the data they process day in and day out is safe and secure:
- They identify unsafe data access or practices in pipelines that could be a source of information leak or policy violation.
- They monitor, log, and track access to databases, data repositories, machines, containers, and code and processing systems and make sure only those who should have access have it. In Dataiku, fine-grained or custom access control can be defined by role.
- They make sure sensitive data is protected. In Dataiku, authentication management enables easy management of users and groups and enterprise-grade security allows all actions in Dataiku to be tracked and monitored using an audit trail.
- Make security tests part of tests and development cycle through a DevSecOp methodology
There is steady demand for the data engineer role, as with the rise of data comes an increased need for efficient management, oversight, and monitoring of data retrieval, storage, and distribution throughout the enterprise. Instead of focusing on data governance (which is more focused on how data is used by authorized users and services), data engineers play a vital role in security governance to help prevent threats of leak or loss.