The ABCs of DevOps (Part I)
Have you ever come across the terms DevOps or MLOps while working on a software development or data science project? If so, you may have wondered what they are and why they’re important. You might also be curious about how to use DevOps in your current project.
This blog is designed to increase your understanding of the DevOps and MLOps domains and how large companies effectively use them in their projects. Also, the blog is written in easy-to-understand language, making it beginner-friendly. I have included examples wherever necessary, and the contents are structured to help you quickly find what you’re looking for. Let’s take a quick look at what this blog covers.
- Introduction to DevOps
- DevOps Pipeline
- Continuity Matters
As this is an introductory blog post, I will keep it brief and straightforward. The more complex topics will be covered in subsequent parts or series of this blog. Follow me here to stay tuned for future updates. :)
Introduction to DevOps!
The main idea behind DevOps is to improve the collaboration and communication between the development and operations teams in software development. Traditionally, the development team would create the software, and then the operations team would deploy and maintain it. However, this approach can lead to delays, errors, and miscommunication between the two teams.
DevOps = Software Development + Operations
DevOps brings together the development and operations teams to work together throughout the entire software development process. This collaboration helps to streamline the process and ensure that everyone is working towards a common goal.
Let me give you an example to help you understand better -
Imagine a small software development team working on a new mobile app for a startup company. The developers are working quickly to build out the features and functionality of the app, while the operations team is focused on setting up the infrastructure for deployment. However, the two teams are working in isolation, without communicating or collaborating effectively.
When the app is launched, it becomes immediately clear that there are performance issues and user complaints about slow loading times and crashes. The development team identifies some bugs and makes fixes, but deploying the updated version is a slow and difficult process due to a lack of coordination and automated deployment processes.
As a result, the updated app is not delivered to users for several days, leading to further frustration and lost revenue for the startup.
DevOps Pipeline
A DevOps pipeline, also known as a continuous delivery pipeline or release pipeline, is a series of automated steps that a software application goes through, from development to deployment.
The DevOps pipeline is designed to ensure that each change to the application is thoroughly tested and validated before it is deployed to production.
As I have described already, DevOps is a practice that brings together software development and operations. During the development stage, software developers write, test, and debug code. On the other hand, in the operations stage, the developed software is deployed to production and monitored to ensure efficient performance.
Let’s now talk about the various pipeline stages of DevOps!
Plan Phase (Dev)
The Plan phase is the initial stage of the DevOps lifecycle, during which teams plan and prioritize the work that needs to be done. It’s the phase where a Product Manager or Project Manager proves their worth.
The requirements related to development will be extracted from the stakeholders and customers to build the product roadmap which will serve as a guide for future development activities.
In short, this phase includes the →
- Defining the project scope,
- Determining the requirements,
- Setting the goals and objectives
- Creating a roadmap to guide the development process.
Some of the activities that take place during the Plan phase include:
- Defining the project scope: The project scope outlines the boundaries and constraints of the project, including the features and functionality that will be included.
- Determining the requirements: The requirements specify the functional and non-functional requirements of the software, including user stories, use cases, and acceptance criteria.
- Setting goals and objectives: The goals and objectives define what the project is intended to achieve, including business outcomes, customer satisfaction, and other key performance indicators (KPIs).
- Creating a roadmap: The roadmap provides a high-level view of the development process, including the timeline, milestones, and key deliverables.
Example:
One example of the Plan phase in action might involve a team developing a new e-commerce platform.
During the Plan phase, the team might work on defining the project scope by specifying the features that the platform should include, such as a shopping cart, product search functionality, and secure checkout.
They might also determine the requirements for the platform, such as the need to support multiple payment options and integrations with third-party services.
Required Tools:
- Microsoft Office, Google Documents
- Microsoft Project (Project Management Tools)
- JIRA, Asana, Mantis (Task Management tool)
During the Plan stage, the product roadmap is created by gathering feedback and requirements from stakeholders and customers. The product roadmap is then broken down into smaller units such as Epics, Features, User Stories, and Backlogs of tasks based on the requirements.
A ticket management system such as Jira, Azure DevOps, or Asana can be used to record and track the product roadmap. These tools offer a variety of features that aid in monitoring project progress, milestones, and issues. The tasks listed in the Backlog are used to plan the sprint and allocate them to teams to kickstart the development.
Before the development team starts writing code, the tasks are prioritized and organized into sprints. A sprint is a short, time-boxed period during which a team works to complete the tasks allocated to them. Typically, a sprint lasts between one and four weeks, and each sprint is focused on delivering a specific set of features or improvements.
During the sprint planning, the development team will evaluate each task in the Backlog, estimate the time and resources required to complete it, and prioritize it based on its importance. The team will then allocate the tasks to individual team members, considering their skills, expertise, and workload.
Once the sprint has begun, daily meetings or stand-ups are conducted to track progress, identify and address any obstacles or issues, and ensure that the project is on track. This allows the team to stay in sync and adapt quickly to any changes or challenges that arise during the development process.
In summary, the Plan stage is critical in establishing the roadmap for the project, breaking it down into manageable tasks, and prioritizing them for allocation to the development team during the sprint planning.
Code Phase (Dev)
The Code phase is an essential stage in the DevOps lifecycle where development teams write and build the code for the software application. This stage involves designing, coding, testing, and integrating the application code.
During the Code phase, the development team uses various coding languages and tools to write the application code that meets the requirements defined in the Plan phase.
Once the code is written, the team will test it thoroughly to ensure it is functional and free of bugs before moving on to the next stage.
The Code phase may involve the following activities:
- Designing the application architecture: The team needs to design the application architecture, including the framework and the software stack required to run the application.
- Writing code: Once the application architecture is designed, the development team starts writing the code for the application.
- Code testing: The team tests the code to ensure it is functional and free of bugs. This includes unit testing, functional testing, integration testing, and performance testing.
- Code review: The team performs code reviews to ensure code quality and maintainability.
- Integration: Once the code is tested and reviewed, the team integrates it with other components of the software application.
Example:
Let’s say a development team is building a mobile app for a new ride-sharing service. In the Code phase, the team would use programming languages such as Java or Swift, as well as tools and frameworks such as Android Studio or Xcode, to write the code for the app. They would also use development methodologies such as Agile or Scrum to manage their work and ensure that the code is delivered on time.
Once the team has written the code, they would then move on to testing it. They would use testing frameworks such as Appium or Espresso to perform unit tests, integration tests, and end-to-end tests to ensure that the app functions as intended and is free of bugs.
After the testing phase, the team would then perform code reviews to ensure the quality and maintainability of the code. They would use code review tools such as GitHub or Bitbucket to collaborate with other team members and ensure that the code is clean, efficient, and adheres to best practices.
Required Tools:
Some tools that can be used during the Code phase are:
- Code editors such as Visual Studio Code, Atom, or Sublime Text
- Version control systems such as Git, SVN, or Mercurial
- Code review tools such as GitHub, Bitbucket, or GitLab
- Testing frameworks such as JUnit, Selenium, or Jest
- Continuous integration and delivery (CI/CD) tools such as Jenkins, Travis CI, or CircleCI
Build Phase (Dev)
The Build phase in the DevOps lifecycle is where the development team actually starts building the software. The software code is written, tested, and prepared for deployment in this phase.
The first step in the Build phase is committing the code to a shared repository. The developer writes the code and commits it to a code repository like Git or SVN. Once the code is committed, the developer creates a pull request to merge the new code with the shared codebase.
Then, another developer reviews the code changes to ensure that there are no issues. The review process is quick and lightweight, but it’s effective in identifying potential issues early. If the reviewer approves the code changes, the code is merged with the shared codebase. If there are issues, the reviewer rejects the code changes, and the developer is notified to resolve the issue.
In parallel, an automated build process is triggered, which builds the codebase and runs a series of tests to identify any issues. These tests include end-to-end, integration, and unit tests. If the build fails or any of the tests fail, the pull request fails, and the developer is notified to fix the issues.
By continuously checking code changes into a shared repository and running builds and tests, integration issues can be minimized, and breaking bugs can be identified early in the development lifecycle. This approach ensures that the software is always in a deployable state, and the team can deploy it to production with confidence.
Example:
In the previous Code phase example, the development team built a ride-sharing app. During the Build phase, the team would use build tools such as Gradle or Xcode to compile the source code and create an executable version of the app that can be installed on a mobile device.
Additionally, the team may also use tools to automate the build process, such as Continuous Integration and Delivery (CI/CD) tools like Jenkins or CircleCI. These tools can automatically compile the code and create new builds of the app whenever changes are pushed to the source code repository.
Required Tools:
Some tools that can be used during the Build phase are:
- Build automation tools such as Apache Maven, Gradle, and Ant
- Continuous Integration and Delivery (CI/CD) tools such as Jenkins, Travis CI, or CircleCI
- Containerization tools such as Docker and Kubernetes
- Artifact repository tools such as Nexus and JFrog
Test Phase (Dev)
After a successful build, the code is automatically deployed to a test environment where it undergoes more thorough testing outside of the development environment.
The Test phase in the DevOps lifecycle is where the application is tested thoroughly to ensure that it meets the desired quality and functional requirements. The primary goal of this phase is to identify and fix any issues or bugs before the application is deployed to production environments.
After the application is deployed to the testing environment, a series of tests are conducted to ensure the software meets the necessary requirements and works as expected.
These tests can be both automated and manual. In manual testing, the application is used by individuals in a way that simulates how a customer would use it (UAT). This allows for any issues or improvements to be identified before the software is deployed to production.
These tests ensure that the application is performing as expected, that all features are working as intended, and that the application is stable and secure.
Required Tools:
- Selenium
- Load testing tools such as Apache JMeter or Gatling for performance testing.
- Bug tracking and management tools such as JIRA or Trello to manage and track bugs and issues during testing.
- Emulators or real devices for testing the application on different platforms and devices.
Release Phase (Ops)
Once the testing has been successfully completed, the build is released as RC (Release Candidate) which is ready for being deployed in production.
In this phase mostly automation kicks in. Sometimes the organization may choose to automate the deployment of any build that passes the build phase. Additionally, developers can leverage feature flags to disable new features that are not yet ready for release to customers. This approach represents the pinnacle of DevOps and enables organizations to deploy numerous releases of their products daily, resulting in rapid time-to-market and continuous improvement of their products.
Required Tools:
Some tools that can be used in the Release phase include:
- Release Management tools such as JIRA or Trello to manage the release process and track any issues or bugs.
- Monitoring tools such as Prometheus or Grafana to monitor the application’s performance and usage.
Deployment Phase (Ops)
The Deploy phase is the phase where the application is actually deployed to the production environment, making it available to the end-users. This phase is closely tied to the Release phase, which is where the application is packaged and prepared for deployment.
Example:
Here’s an example of the Deploy phase for a ride-sharing app:
- Build the Deployment Package: Before deploying the application, the development team will build a deployment package that contains all the necessary files and configurations for the application to run. This package may be a Docker container or a package created with tools like Ansible, Chef, or Puppet.
- Configure the Environment: The development team will ensure that the production environment is configured correctly to run the application. This may involve setting up load balancers, configuring network settings, and installing any necessary software.
- Deploy the Application: Once the deployment package is built and the environment is configured, the development team can deploy the application. They may use tools like Kubernetes, Docker Swarm, or AWS Elastic Beanstalk to manage the deployment.
- Rollout Strategy: To ensure that the deployment is successful, the development team will follow a rollout strategy. This may involve deploying the application to a small group of users initially and gradually increasing the user base as the application stability is confirmed.
- Monitoring and Feedback: After the deployment, the development team will monitor the application and gather feedback from end users to identify any issues or bugs that may arise. They may use monitoring tools such as Prometheus or Grafana to monitor the application’s performance and usage. They will also gather feedback from end-users through feedback channels such as in-app feedback, social media, or customer support.
By using these tools and techniques, the development team can ensure that the ride-sharing app is deployed to end-users in a controlled and efficient manner. They can also ensure that any issues or bugs that may arise during the deployment are identified and addressed promptly, leading to a better user experience and improved adoption of the application.
Required Tools:
Some tools that can be used in the Deploy phase include:
- Configuration Management tools like Ansible, Chef, or Puppet to configure the environment.
- Containerization tools like Docker to build and deploy the application.
- Container orchestration tools like Kubernetes or Docker Swarm to manage the deployment.
- Monitoring tools like Prometheus or Grafana to monitor the application’s performance and usage.
Operate Phase (Ops)
In this stage, we know that the build is deployed and it is ready to be used. In this phase, the operation teams make sure that things run smoothly as expected. They also take care of server configuring and provisioning using tools like Chef.
Monitor Phase (Ops)
The Monitor phase of DevOps is the final stage in the DevOps cycle, but it is by no means the least important. In fact, monitoring is crucial for ensuring that the application runs smoothly in production and that any issues are identified and addressed as quickly as possible.
During this phase, the operations team is responsible for monitoring the application and the infrastructure that supports it. This includes monitoring the servers, databases, network, and any other components that the application relies on. By monitoring these components, the operations team can identify any potential issues before they impact the end users.
To achieve effective monitoring, it is essential to have the right tools in place.
These tools can include:
- Monitoring Tools: There are a variety of monitoring tools available that can be used to monitor the application and infrastructure. These tools can range from essential tools like Nagios, Zabbix, and Prometheus to more advanced tools like Datadog, New Relic, and Dynatrace.
- Logging and Analytics Tools: Logging and analytics tools can be used to gather data on the application and infrastructure. This data can then be used to identify trends and patterns that can help to identify issues before they become critical. Examples of logging and analytics tools include ELK Stack, Splunk, and Graylog.
- Alerting and Notification Tools: Alerting and notification tools can be used to notify the operations team when issues arise. These tools can be configured to send notifications via email, SMS, or other messaging systems. Examples of alerting and notification tools include PagerDuty, OpsGenie, and VictorOps.
- Dashboard and Visualization Tools: Dashboard and visualization tools can be used to provide a real-time view of the application and infrastructure. These tools can be used to monitor key metrics such as CPU usage, memory usage, network traffic, and more. Examples of dashboard and visualization tools include Grafana, Kibana, and Tableau.
During the monitoring phase, it is important to define and monitor key performance indicators (KPIs) relevant to the application and business goals. These KPIs can include metrics such as uptime, response time, error rate, and user engagement. By monitoring these KPIs, the operations team can identify any issues that may be impacting the end users and take action to resolve them.
Continuity Matters!
You probably heard about these words somewhere → “Continuous Integration / Continuous Delivery / Continuous Deployment / Continuous Feedback” etc. These concepts float around the DevOps pipeline. Therefore I thought you should be familiar with these concepts.
Continuous Integration
Continuous Integration (CI) is a software development practice where developers frequently merge their code changes into a central repository, and automated builds and tests are run to check for any integration errors.
This is how it works →
- Developers write and save their code changes to a shared code repository, like GitHub or Bitbucket.
- A CI tool (such as Jenkins, Travis CI, or CircleCI) monitors the code repository for any changes. When it detects new code changes, it pulls the latest code from the repository.
- The CI tool builds the code by compiling it into an executable program. This ensures that the code is free of syntax errors and other common mistakes.
- The CI tool runs a suite of automated tests to verify that the code works as intended and does not introduce any new bugs or errors.
- If any errors or bugs are found, the CI tool reports them back to the developers, who can then fix them and repeat the process until the code is working properly.
- Once the code passes all tests, it is considered “clean” and can be deployed to production.
In summary, CI helps to improve software quality and accelerate the development process by automating the process of building and testing code changes. By catching errors early on and ensuring that code is always in a stable, working state, CI makes it easier and faster to deliver high-quality software to end-users.
Continuous Delivery
Continuous Delivery (CD) is the practice of automating the entire software delivery process so that code changes can be delivered to production quickly and reliably. Consider it an extension of Continuous Integration (CI).
Here’s an example to help explain how it works:
Let’s say you’re part of a team working on an e-commerce website. You’ve made some code changes to the checkout process to make it more user-friendly. Now you want to deliver those changes to production so that customers can start using them.
Here’s how Continuous Delivery can help:
- You commit your code changes to the shared code repository.
- The Continuous Integration (CI) system runs tests on your code changes to ensure that they’re stable and error-free.
- If the tests pass, the CD system kicks in and automatically packages your code changes into a release candidate.
- The CD system then deploys the release candidate to a staging environment, which is a production-like environment used for testing.
- Automated tests are run on the staging environment to ensure that everything works as expected.
- If the tests pass on the staging environment, the CD system deploys the release candidate to production.
- The CD system then monitors the production environment for any errors or issues and rolls back the changes if any issues are detected.
In this example, Continuous Delivery ensures that your code changes are delivered to production quickly and reliably, while also minimizing the risk of errors or issues. It also enables your team to release changes to production more frequently, allowing you to deliver value to your customers faster.
Continuous Deployment
Continuous Delivery (CD) and Continuous Deployment (CD) are both practices that are often used in conjunction with Continuous Integration (CI) to automate software delivery. While they are similar in some ways, there is a key difference between the two:
Continuous Delivery (CD) is the practice of automating the entire software delivery process so that code changes can be delivered to production quickly and reliably, but it still requires a human decision to be made to deploy the code changes to production.
In contrast, Continuous Deployment (CD) is the practice of automatically deploying code changes to production as soon as they pass all tests in the CD pipeline. In other words, the CD takes automation one step further by eliminating the need for a human decision to deploy the code changes.
To illustrate the difference, let’s take the example of an e-commerce website where a team has made some code changes to the checkout process. With Continuous Delivery, the CD system packages and deploys the code changes to a staging environment where automated tests are run. If the tests pass, a human makes a decision to deploy the changes to production. With Continuous Deployment, however, the CD system automatically deploys the changes to production without the need for a human decision.
In summary, the key difference between Continuous Delivery and Continuous Deployment is that Continuous Delivery still requires a human decision to be made to deploy the code changes to production, whereas Continuous Deployment automatically deploys the changes without human intervention once they pass all tests in the pipeline.
Continuous Feedback
Continuous Feedback is a key component of the DevOps process that involves continuously gathering and analyzing feedback from users and systems to inform ongoing software development efforts.
Here’s an example to help explain how it works:
Imagine that you’re part of a team working on a mobile app that helps people track their fitness goals. Here’s how Continuous Feedback can help improve the app:
- The team releases a new version of the app that includes a feature to track water intake.
- The app is released to a subset of users, who are asked to provide feedback on the new feature.
- The team uses analytics tools to track how users are interacting with the app, including how often they use the new water-tracking feature.
- Based on user feedback and analytics data, the team identifies areas for improvement and makes changes to the app, such as adding more intuitive controls for the water tracking feature.
- The team releases a new version of the app that includes the changes, and the process starts again with gathering feedback on the updated version.
By continually gathering and analyzing feedback from users and systems, the team is able to make data-driven decisions about how to improve the app. This process helps ensure that the app meets the needs of its users and stays relevant over time.
Continuous Feedback is not just limited to user feedback. It can also involve gathering feedback from systems, such as monitoring tools that track the performance and stability of the app. This feedback can help the team identify and address issues before they impact users.
Overall, Continuous Feedback is a crucial part of the DevOps process that helps teams deliver high-quality software that meets the needs of its users. It enables teams to continually learn and improve based on feedback from users and systems, leading to better software and happier users.
Footnote
I am grateful for your interest in perusing this introductory blog post and trust that it has proven to be both captivating and enlightening. It gives me great pleasure to have been able to provide you with valuable educational content.
It is essential to note that this blog post serves as a precursor to an upcoming series of blogs that explore more intricate topics such as Dockers, Jenkins, and Travis CI. These topics hold a central position in the realm of DevOps and possess the potential to make a significant impact on software development and delivery processes.
I encourage you to follow me closely so that you may stay informed of any new updates or releases in the series. My unwavering commitment to presenting content that is informative and accessible ensures that this series will appeal to both beginners and experienced professionals alike.
Let’s get acquainted with each other on LinkedIn.
Tadaaa!
