First Steps DevOps

Tags:
2022 12 01 head

The DevOps movement started to gather momentum with the first DevOps conference held in Ghent, Belgium in 2009 [1].

The approach aims to shorten the system development life-cycle and to provide continuous delivery with high software quality. DevOps is complementary with agile software development approaches.

Several DevOps aspects came from the Agile method.

Seminal books [1, 2, 3, 4] and good practices were published around the same time.

The DevOps approach belongs to the regular instruments of professional software developers. Major development platforms, GitHub, GitLab, Bitbucket and Microsoft Azure DevOps do support continuous integration, delivery, and deployments for years.

The DORA metrics define a standard approach to measure the quality and efficiency of a DevOps or agile process implementation in an organization. They were published in 2016. The metrics for throughput are deployment frequency and lead time for changes. The metrics for stability are mean time to recover and change failure rate.

They define a standard approach to measure the quality and efficiency of a DevOps or agile process implementation in an organization. An agile team shall be able to deploy a version of their digital platform multiple times per day. Frequent and economical deployments require the full automation of the integration, delivery, and deployment steps.

Below, we provide a detailed possible approach to deploying DevOps concepts in your organization. These recipes are helpful for companies with little knowledge and experience in agile and DevOps approaches for digital product development [5].

Continuous integration

In software engineering, continuous integration CI is the practice of merging all developers' working copies to a shared mainline several times a day [2].

The workflow is:

  1. Write some code, compile and run test locally using a script.
    No manual activities shall be required to compile the solution and run all tests.

  2. Commit, push, and compile code in the continuous integration environment.
    A continuous integration pipeline shall be available. The environment is a clean new one each time the pipeline is triggered.

  3. Run static code analyzers, unit tests, and integration in this integration environment.

  4. Optionally, trigger continuous delivery if the earlier steps were successful.

Classical questions and answers FAQ are:

  1. How often shall a developer push new code to the shared repository?

    Laggard developers should push multiple times a week. Professional developers push multiple times a day [4].

  2. What steps are part of a continuous integration script?

    The script shall at least compile the solution. Static code checkers shall be called to validate the changes and ensure minimal code quality.
    Modern checkers detect syntactic, semantic, or security violations. OWASP rules are readily available to start implementing DevSecOps. Unit tests and if reasonable integration tests shall be executed automatically to verify the changes.

  3. When should the continuous integration script be executed?

    Each time a git push is registered and the repository is updated [3]. You want to guaranty that the repository is in a consistent state.
    Any developer should be able to check out the repository and start working.

  4. Can you add manual steps to your continuous integration script?

    No, manual steps are the death of continuous integration and automation. It is called continuous because it is automated.

Continuous delivery

Continuous delivery CD is a software engineering approach in which teams produce software in short cycles. It ensures that the software can be reliably released at any time and, when releasing the software, without doing so manually. It aims at building, testing, and releasing software with greater speed and frequency.

The approach helps reduce the cost, time and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

Classical questions and answers FAQ are:

  1. How can a new delivery package be generated?

    Your continuous integration pipeline should have a delivery step and produce an executable artifact. Your hosting platform provides a service to upload versioned artifacts to a repository. Java artifacts are often stored in Maven Central. Docker images are often uploaded to Docker Hub.

  2. Where should we store the migration scripts?

    Migration and configuration scripts are part of the delivery. The installation process shall perform an unattended migration of the configuration and databases. Manual steps are not allowed.

  3. When is a new delivery package created?

    Upon completion of compilation, static verification, unit tests, and integration tests, you should create the delivery package.

  4. Which preferred format should you use?

    For complex solutions, the preferred format is a docker image or a kubernetes pod.

  5. How do you identify a specific version of a package?

    Semantic versioning is the standard approach to generating legible package identifiers. You are free to adopt another naming scheme for software versioning. Document your approach.
    Implement a function so that your users can find out the version of the product they are using.

Continuous deployment

Continuous deployment CD is a software engineering approach in which software functionalities are delivered frequently through automated deployments. Continuous deployment contrasts with continuous delivery. It is a similar approach in which software functionalities are also frequently delivered and deemed to be potentially capable of being deployed.

  1. Are manual steps good practice for continuous deployment?

    Never perform manual steps. Continuous activities require full automation. Manual steps can only be performed if a trained collaborator is available. Manual activities are expensive, automated activities are cheap.

  2. What should you do if a deployment fails?

    The environment should perform an automatic rollback to the last working configuration. It is the reason why DORA metrics [1] [2] measure mean time to recover and change failure rate.

  3. Are commercial platforms available for continuous deployment?

    Major cloud services Amazon AWS, Google Cloud, Microsoft Azure, Heroku, etc. provide continuous deployment pipelines and integration with product development platforms.

Logging and Monitoring

Site reliability engineering and DevOps practices are standard approaches in any professional organization running digital products productively.

Monitoring is used as a PDCA loop to continuously improve the construction and delivery process of a productive solution.

Classical questions and answers FAQ are:

  1. What should be logged?

    The software architecture document of your solution should document how and what information you are logging. Mapped diagnostic contexts are used to track session and user related actions and log records [4] [5].

  2. Should logging information be archived?

    The corporate governance guidelines shall prescribe how long logging information is stored and who can access the data.
    Logging information should be available to protect your company against legal complaints. The information is also invaluable for security incident analysis [6].

  3. Should applications be monitored?

    All productive installations should be monitored through a professional solution. The monitoring approach should detect issues before they impede user processes. You shall document the alarming and recovery process. [7]

GitHub, GitLab, BitBucket

Modern software development departments use these platforms for collaboration and implementation.

2022 12 01 ci vs cd

As stated on the GitHub website, we have the following definitions.

Automation is a core principle for achieving DevOps success and CI/CD is a critical component. CI/CD consists of continuous integration and continuous delivery or continuous deployment. Put together, they form a CI/CD/CD pipeline. Automated workflows that help DevOps teams cut down on manual tasks:

  • Continuous integration (CI) automatically builds, tests, and integrates code changes within a shared repository.

  • Continuous delivery (CD) automatically delivers code changes to production-ready environments for a final approval.

  • Continuous deployment (CD) automatically deploys code changes to customers directly.

Multiple Toolchains

  • GitHub supports multiple toolchain installations through successive steps in the pipeline.

  • BitBucket can only install one toolchain for a pipeline. Additional toolchains must be installed using package management of the operating system for your Docker image. Beware that package managers are dependent of the used distribution. The openjdk regular image uses, for example, microdnf as a package manager.

Actual Java JDK Releases

  • GitHub does not support the latest official Java releases on time.

  • Bitbucket supports all officially released Java versions.

You need a platform and automation expert in your development team to harvest the advantages.

odoo.sh has selected GitHub as their preferred platform. We shall use the GitHub CI/CD/CD approach as defined in odoo.sh to develop and deploy all our application development.

GitHub provides initial support for Security, DevOps, and agile approaches. We shall evaluate these tools and, if relevant, use them as standard tooling for our Odoo software tailoring and development.

The CI tools of GitHub are ideal to implement DevSecOps approaches on odoo modules and internal Python application development. The Sonar scanners verify, for example, compliance to OWASP vulnerabilities. This verification step can be executed on a repository update to remove vulnerabilities in deployed oddo instances.

Legacy System Variations

Legacy systems must compromise how they implement their pipelines. Source code shall always follow the clean code and DevOps principles. Configuration information stored in the database must often be migrated between staging areas. Please document your automated process and rules.

Modern systems have fully adopted the configuration and infrastructure as code approach. Migrate your approach to automated configuration scripts to configure and initialize your products.

Incremental migration of database schema and productive data is a well-understood software engineering problem [5] for decades. Tools such as FlyWay or Liquibase provide enterprise grade productive solutions.

The open ERP Odoo platform uses GitHub to offer a clean DevOps process for the Odoo.sh cloud platform. The approach has two components:

  • Source code and modules updates follow clean code and DevOps principles. You create new functions in the development branches, merge them in staging branches to test them, and release the changes to the production branch for deployment. The approach is like the staging process of the Linux kernel.

  • Database updates are performed slightly differently between development and the other staging area. The staging area either gets a copy of the production database or a migrated version of the previous staging area database. This behavior can be configured.

Merging a staging branch in the production branch only merges the source code. Any configuration changes you made in the staging databases are not passed to the production database.

If you test configuration changes in staging branches, and you want them to be applied in production, you have to either:

  • Write the configuration changes in XML data files overriding the default configuration or views in your branches, and then increase the version of your module in its manifest (manifest.py) to trigger the update of the module when you merge your staging branch in your production branch. This is the best practice for a better scalability of your developments as you use the Git versioning features for all your configuration changes. Therefore, you have traceability for your changes. This approach is compatible with clean code and DevOps.

  • Pass them manually from your staging to your production database, by copying and pasting them. This approach is probably a motive to fire the developer because traceability, auditability, and reproducibility are not enforced. Regular corporate governance rules are neglected and could have legal consequences in case of incidents on the production server.

When you want to test the changes of your development branches with the production data, you can either:

  • Merge the development branch into your staging branch by dragging and dropping it onto the desired staging branch.

  • Drag and drop the development branch on the staging section title, to make it become a staging branch.

When your latest changes are ready for production, you can merge the staging branch onto your production branch to deploy in production your newest features.

The above process is documented for Odoo version 15 under the branches section.

Unprofessional Odoo Approach

Beware that the odoo.sh environment does not hinder amateur developers from doing foolish things with git.

The quote below is from the official website of Odoo.sh. The statement violates all the rules of clean code, DevOps, and professional software development.

I wishfully hope the Odoo organization will mature and remove these misleading statements.

If you are bold enough, you can merge your development branches into your production branch as well.
It just means you skip the validation of your changes with the production data through a staging branch.

— odoo.sh
2022

Professional Odoo Approach

On the bright side, odoo.sh supports and advocates the following clean code compatible workflow:

  1. Source code is developed and tested in the development branch. Good practice is to write unit tests and have demonstration data. Each time you push changes, a clean instance is created, the demonstration data is loaded, and all unit tests are executed.
    odoo.sh good practices implement TDD, ATDD, and quality gates as expected from a professional approach.

  2. Once you are successful, you merge your development branch to the staging branch. The environment will automatically provide a copy of the production database in the staging area. You can test your new functions with production data without impeding the production server.

  3. Once you are satisfied with the behavior of the new application with the copy of the production data, you can merge to production. A new production server will be deployed. If the deployment is unsuccessful, it will be rolled back to minimize downtime and to improve mean-time to recover.

Tips and Tricks

Professional developers perform only code changes in the development branches. A development branch is often a feature branch.

Staging and production branches are effectively deployment triggers in the odoo.sh world. Staging branches shall only be used to validate a new version of your product.

Logically, only one production branch is available.

odoo.sh abuses git branches as triggers for deployment to staging and production. This approach could historically be justified due to the clumsy way database changes were done in the past.

Migration scripts shall be robust and test their constraints [8]. For example, if the script expects the existence of a value or an identifier in the database, it should test this integrity rule and abort if an error happens. The goal is to avoid inconsistent states after an update or a migration.

The new good practice is finally compatible with clean code, infrastructure as code, and DevOps. This approach is promoted in the official odoo.sh documentation. [9].

As stated by Odoo, you shall write the configuration changes in XML data files overriding the default configuration or views in your branches. This is the best practice for a better scalability of your developments as you will use the Git versioning features for all your configuration changes.

Lessons Learnt

Clean code and DevOps good practices are around for almost two decades. Senior developers in your team should have read the seminal books in this area. These experienced senior designers shall coach younger team members.

The InfoQ DevOps and cloud graph show DevOps themes investigated in companies during 2022. Your experts shall apply most or all the approaches documented in the two columns on the right side.

2022 12 01 devops cloud graph

A professional organization shall have a continuous formation and training path for all collaborators. I currently often have to answer basic questions about DevOps, clean code, or Scrum. Quite a few developers do not know basic definitions or how to use Git or GitHub. It is time to read the documentation.

Please train and read the documentation. Do not be a noob.

We politely ask you to read books and documentation. Avoid receiving an RTFM injunction.

You need expertise in Docker, Kubernetes and in your development platform [3]. We recommend initially using GitHub. It is the most used platform. Most developers are versed in how to use the provided features.

If your company is missing experienced DevOps and agile professionals, it is time to improve your recruitment process [5].

All development platforms have extensive support for continuous integration, delivery, and deployment. DevOps approaches are well-supported. Often, advanced concepts such as monitoring, DevSecOps, or license management are supported. Extensive tutorials and good practices are available.

Use the provided tools and approaches to professionally develop and operate your digital platforms.

Links

References

[1] N. Forsgren, J. Humble, and G. Kim, Accelerate: The Science of Lean Software and DevOps. IT Revolution Press [Online]. Available: https://www.amazon.com/dp/B07B9F83WM

[2] G. Kim and N. Forsgren, DevOps Handbook, Second. IT Revolution Press, 2021 [Online]. Available: https://www.amazon.com//dp/B09G2GS39R

[3] G. Kim, The Phoenix Project a novel about IT, DevOps, and helping your business win. IT Revolution Press [Online]. Available: https://www.amazon.com/dp/0988262592

[4] B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Site Reliability Engineering. O’Reilly Media, Inc., 2016 [Online]. Available: https://www.amazon.com/dp/1492029505

[5] A. Scott and P. J. Sadalage, Refactoring Databases. Addison-Wesley Professional, 2006 [Online]. Available: https://www.amazon.com/dp/B001QAP36E

1. Development organizations had around fifteen years to practice DevOps and adapt their application lifecycle management processes.
2. Grady Booch first proposed the term Continuous Integration CI in 1991. Analyze your git logs and find out if your developers are truly merging working copies to a shared mainline several times a day. Laggard programmers use git as a backup tool and not as a collaborative development platform.
3. You can either use trunk based development or configure the build trigger for all your feature branches.
4. odoo.sh provides logging support for a deployed solution. Corporate governance often requires that the log files be archived in a separate location. These files are required when a forensic analysis of a security situation shall be investigated.
5. odoo has an audit module supporting auditing of critical actions initiated through user interactions or external systems. The audit module is somewhat primitive. The audit configuration is performed at the database level and not at the functional level. The configuration responsible needs deep understanding of which modules perform specific operations on the database.
6. The logging files and audit traces shall be stored on company premises to fulfill legal and corporate governance rules.
7. odoo.sh provides monitoring for deployed solutions. The monitoring is currently based on Kibana. All productive servers shall be monitored with a similar approach. The monitoring approach shall be tuned based on production incidents.
8. If you write, for example, an SQL database schema migration script, you use clauses such as DROP IF EXISTS.
9. Manual editions through copy and paste are a violation of sound engineering development practices. You lose reproducibility, versioning of a single source for truth, and auditability. Assume that you violate development guidelines and corporate governance.