DevOps vs. DataOps: What’s the Difference and Which Do You Need as Data Analyst and Engineering Consultant?

What is DevOps?

DevOps is short for Development and Operations. Previously, software engineers and IT worked separately, causing confusion, and released products were filled with bugs.

With DevOps, both software engineers and IT work together to ensure everyone is on the same page, resulting in faster development and release of products.

DevOps relies on a CI/CD pipeline, which is used to test software changes before releasing the product.

What is DataOps?

Inspired by DevOps, DataOps follows a similar structure to ensure that the merging and transforming of data is as smooth as possible. Just like in DevOps, DataOps uses version control to track changes and the CI/CD automation to ship these changes to production.

Components of DevOps/DataOps (and their differences):

Plan: Decide which features to implement and how to assess its performance.

Code: As the name suggests, code out the features, but make sure to have your code reviewed!

Build: Assemble all the components into a package.

Test: Begin testing to assess the quality of the changes made before release.

Release and Deploy: Release the product for your customers.

Operate and Monitor: Observe logs and monitors to ensure smooth operations of the products being released.

This cycle would continue as long as the product is maintained.

Plan: Decide which data to modify or create a new data source. Make sure to define KPIs!

Develop: Create data pipelines and perform data transformation. Submit changes for code review.

Test: Test the data to ensure that modifications are running correctly.

Deploy: Move the changes from development to production.

Operate and Observe: Make sure data changes are working as intended. Ensure access to data is controlled by RBAC (role-based access control).

Discover and Analyze: Stakeholders are able to use the data to find insights.

Personas (people involved in DevOps):

Developer (Software engineers)

IT administrator

Business Stakeholder

Personas:

Engineers

Analysts

Decision Makers

Notice how the personas are broader in the DataOps section! DataOps recognizes that there are so many roles and that these roles can differ from person to person, so it makes the personas as broad as possible.

What does this mean to a Data Analyst and Engineering Consultant?

While it is nice to know both and they certainly overlap in terms of processes, learning about DataOps is more crucial for Data Analysts and Engineers (assuming you are one or aspiring to be one). It is our responsibility to ensure that the data is cleaned and modified in a way where all changes are monitored and logical. We are deeply concerned about the quality of data and must ensure that all changes are working as intended.

Source and Images obtained from: https://www.getdbt.com/blog/dataops-devops-difference

Author:

Luis Tepoz Vazquez

View Profile