DevOps: Infrastructure Standardization or You Build It, You Run It?

The DevOps principle of “you build it, you run it” emphasizes the continuous ownership of the development team throughout the application lifecycle. This not only promotes agile responses to problems and accelerates the development process, but also positively influences the mindset of team members. However, complex enterprise software often faces challenges in the real world. Here, we examine two key challenges in implementing “you build it, you run it” in practice and present solutions that mgm has successfully implemented.

At mgm, we have repeatedly found in our project experience that the challenges in implementing “you build it, you run it” can be traced back to two core issues. In the past, we have only applied the “you build it, you run it” approach when the necessary expertise for all aspects of the process is ensured by appropriate personnel in the project. Unfortunately, this is not always the case.

1. The DevOps Staffing Dilemma

The staffing dilemma is a familiar problem in the industry: DevOps engineers are in short supply, and smaller projects often have to make do without adequate staff. Training alone is usually not enough, as project teams have limited capacity and one-time training is not enough.
Partial implementations or simplistic assumptions should be avoided, as they can lead to significant problems. For example, software that works perfectly in the development environment may not work at the customer site because basic assumptions do not apply outside the development environment.

2. Insufficient cross-project sharing

Our experience has shown that while the DevOps principle of “you build it, you run it” promotes high levels of autonomy and accountability with positive effects on software development, it can also lead to insufficient cross-project sharing. In project reality, previously solved problems are often solved again, and often in different ways. While these problems can be interesting, they often distract from the main goal of creating functional solutions, commonly known as business value.

To address these challenges, mgm focuses on efficiency in enterprise application development through a combination of expertise, our Enterprise Low Code Platform A12, and continuous cross-project exchange.

Our solution: Standardizing the Technical Infrastructure

To address these challenges, mgm set up a competence team for standardized project infrastructure, the TPI team (TPI = Technical Project Infrastructure). Team leader Ralf Brauwers explains:

“We provide projects with standardized infrastructures, cross-project standards and best practices. Through DevOps training and the provision of automatically generated infrastructure, we enable projects to access these resources. Ideally, the entire infrastructure stack is automatically available once the project requirements are clarified at the beginning. This allows projects to implement features from the beginning and test them locally or on the TPI Dev Cluster.”

Cross-project standards provide many benefits:

  • Coherence and consistency: Projects do not have to answer the same questions every time, but can rely on proven best practices.
  • Efficient development: Teams can focus on building functionality.
  • Efficient onboarding: Reduces onboarding time for team transitions because of cross-project conventions.
  • Interoperability: Source code migration between projects is simplified because the infrastructure is standardized.
  • Collaboration: Collaboration and knowledge sharing between teams is facilitated by the cross-functional nature.

There are also benefits from the perspective of the infrastructure teams:

  • Automation: Infrastructure can be automatically provisioned.
  • Patch management: Automation facilitates regular updates, greatly improving security.
  • Incremental changes: Small changes are often less error-prone than large migrations and ensure smoother operations.
  • Infrastructure as Code: Infrastructure configurations are in the form of code, so changes can be tracked and reversed if necessary.
  • Efficiency: With a team of experts and a focus on automated infrastructure provisioning, this can be done very efficiently.

One example:

We recently faced a challenge that required our full attention: a critical bug in the Continuous Integration/Continuous Deployment (CI/CD) system (Jenkins) required quick action. Within hours, we successfully fixed the bug through automated patch deployments to no less than 60 instances.

Implementation Tips and Strategies

However, some project teams have difficulty implementing these standards. “The process doesn’t always fit perfectly with the individual needs of each project,” says Brauwers. In addition, project teams may not be able to adequately respond to evolving standards. This can be especially true when changes are unpredictable, when there are financial constraints, or when changes occur close to project release dates. To overcome this challenge, it is advisable to focus on providing the best possible support for migration processes and to offer flexible timeframes. “The arguments against cross-project standardization are understandable, but there are ways to overcome them,” says Brauwers.

Tip 1: For example, a small budget for infrastructure adjustments should always be included in the project budget from the outset to avoid financial problems. Failure to do so may cause financial difficulties for the project, especially if the project is long, or may require renegotiation.
Tip 2: A classic scenario in projects nearing release is the “breaking change”, a change to a part of the infrastructure that requires adjustments in the project. The competence teams providing the infrastructure should make sure to announce a breaking change in advance and provide sufficient time frames for project migrations. This allows everyone to prepare and make adjustments within that time frame, preventing unwelcome surprises and ensuring planning certainty.

Conclusion

The introduction of a centralized team to provide DevOps services (clusters, build infrastructure, SCM, etc.), along with cross-project standards and best practices, offers clear benefits for software development in large enterprises. It enables project teams to focus on developing functionality and ensures a more efficient and secure development process. The key is to establish common guidelines and address key issues up front to ensure enterprise-wide standardization. De facto industry standards such as SemVer and Gitflows provide proven models, while organizations must develop their own standards for specific requirements such as the kickoff process. This strategic standardization not only promotes consistency and efficiency, but also provides a clear foundation for collaboration and development across the enterprise.