Integration Streams

Integration Streams combine the output of multiple developers, and/or from multiple contributing teams in a complicated or complex system. They also provide a staging mechanism for testing prior to promotion from “build” to “release” enabling progression through the levels of done. Integration Streams are typically implemented as “streams” or “branches” in version control tooling.

In its most basic form, the integration stream provides a mechanism to allow integration of changes to code/components into a cohesive product, providing a point for integration testing that promotes the product to a higher level of completeness (or levels of Definition of Done). Integration streams are tightly bound to Software Configuration Management and the concepts of Builds and Releases.  Modern code management tools (e.g. git) allow individual developers to work, and share numerous branches simultaneously – in these cases multiple Developer Workspaces, and multiple Integration Streams are all implemented as branches.

Developers can promote their changes to integration streams after successfully unit testing their code. This allows for separation of individual developer workspaces, and work on different features, from the integrated/combined team product, providing a way of ensuring the quality of a developer’s changes before they are shared. Once changes are promoted to an integration stream tests can be run to try and identify integration bugs caused by successive merges and to check for regressions.
Often teams will establish a rule that developers do not promote changes to the integration stream unless the changes pass all unit tests locally. Some teams may appoint a build/integration manager who is responsible for pulling changes promoted by developers into an integration stream and ensuring successful merging. When such a rule is established it is often phrased as “You’re not allowed to break the build/integration stream” whereas of course we actually want to detect integration problems as soon as possible so finding a breakage is a good thing. The intent of this rule is actually to state “If the integration is broken we must fix it before continuing”.

A Build is change or collection of changes, integrated into a product, that has had some (typically minimal) level of inspection and quality assurance.

A Build may be performed at an individual personal level or at a team level where changes from more than one team member are integrated into the build. Inspection and quality assurance activities are normally a little more involved at team level as the Definition of Done is a little higher but builds do tend to have a relatively low Definition of Done – not every build will be fully tested. Builds happen frequently, often many times a day.

A Release has a dual nature:

  • a Build, with a high level of Done, packaged for operational deployment that is a viable cohesive product delivering Business Value.
  • a Planning concept (Release Plan), typically containing a number of deliveries produced through, iterations/sprints or continuous flow, that combines a set of “done” high level-requirements, architecture and tests – often represented as a Milestone.

A Release is typically formed once in a Release planning timebox after a succession of internal builds but can be created from every successful team build when Continuous Deployment practices are used. Releases will reach a high Definition of Done, often as far as “End 2 End Tested” but may not necessarily be adopted by the business just because they are ready as Adoption and Business Change cycles may be decoupled from Release cadences.

Layered Integration

It is possible to extend the basic model to provide multiple layers of integration streams. The same rules apply at multiple levels so that promotion of change sets is not allowed unless the changes have passed their basic level of testing. This promotes the identification of defects as early as possible in the development workflow. The use of multiple layered integration streams also supports the clear separation of Definitions of Done and therefore levels of testing.

In simple projects this separates unit from integration and system tests, in more complex scenarios integration streams allows for separation of component, product and system tests at different levels of Done. One of strengths of the H-Model in Holistic Software Development is that levels of requirements decomposition are mapped to levels of architecture, integrated together via separated integration streams and mapped to Definitions of Done. The separation of levels aid identification of bugs in complex systems and speeds up pushing requirements driven integration as high up the levels of Done stack as possible – and as frequently as possible.

The integration stream can initially be implemented with a release cadence allowing the synchronized integration of work from contributing individuals or teams. Often a team will integrate all changes performed during a sprint for a build towards the end of their sprint. This kind of rhythmic integration stream is sometimes called a “Release Train“. However, we recommend that individual teams and multiple teams working together in developing complex products aspire to Continuous Integration as any delay in integration caused by a constituent team not hitting an integration point can cause significant problems if there are cross team dependencies.

We do not recommend synchronizing sprints/iterations and forcing release cadences. Instead we promote continuous integration, pushing every change as high up the Definition of Done stack as possible, independently from other changes. Where changes need coordinating we pull towards Business Value through Integration Scenarios. Synchronization forces everyone to move at the slowest pace and is simply unnecessary. It is often argued for since it makes planning easier, but it doesn’t make software delivery easier.

Continuous Integration refers to the practice of continuously applying quality assurance to software, applying small amounts of effort frequently in contrast to traditional practices of applying a large amount of integration and quality assurance effort at the end of a project, phase or timebox.

Continuous Integration involves building the product repeatedly, as often as after every change in an Integration Stream or woskspace performing some level of quality assurance to establish a basic level of Done such as “Unit Tested”.

The overriding principle is that the integration stream always has a runnable version of the product. This promotes the concepts of rigorous testing, identifies integration issues as they occur and also provides the opportunity for shipment of any chosen product version as determined by the business.

Of course, this pattern can be used recursively for teams who need to integrate their work. Teams can push, or integrators can pull, changes from team integration streams into higher level integration streams, testing integration scenarios prior to creating a release. Which may in turn be integrated into a higher level product and so on. If using recursive layered integration, we recommend that the layers are kept to the absolute minimum as dictated by system-of-system structural complexity.

When building complex systems-of-systems we recommend that the higher levels of integration stream “pull” known good versions, or atomic changes, from the contributing teams. A version or change is not permitted promotion unless it has satisfied pre-determined acceptance criteria. These acceptance criteria may be a combination of tests including a threshold of code coverage (the Quality Confidence measure can be used as a quality gate prior to promotion) but should always include inspection and actually running the code.

Generally speaking, especially when working cross-team environments sharing code changes for reuse and/or integration is not ideal. Instead we recommend using architectural interface dependencies that are preferable to code dependencies. This is why the layered models above, and the Big Picture, show built components (as small color-coded squares) rather than change sets (triangles) being shared.

When sharing via interfaces then we recommend interfaces are versioned and backwards compatible wherever possible. Resist the urge to freeze interfaces early, as that inhibits change and introduces architectural fragility. Instead each published version of an interface should be immutable and long-lived, but expect several to be published over time. Components and services can, and should, support multiple versions of interfaces at the same time.

Implementation with Software Configuration Management

Integration Streams are most effective when their structure aligns with both team structures, requirements levels and therefore Definitions of Done. Similarly the layers of integration streams are an ideal design for SCM structure. Depending on the SCM tool used integration streams may be best represented as branches (git etc.) or directly as streams/repositories.

If sharing binaries at higher levels (preferable to code sharing) then typically a binary repository and dependency management system will be used such as Sonatype Nexus and maven.

Release Planning

A Release Plan is a medium term plan that defines a series of high level goals that delivery teams can “sprint” or “flow” towards.
Release plans join up the Portfolio/Programme views with Projects/Delivery Teams.


When frequently or continuously releasing there is likely to be a need to frequently and/or continuously deploy to live operational environments. The close collaboration required between Operational stakeholders and Development stakeholders to achieve this smooth deployment and ensure correct inclusion of Operational requirements (especially non-functional requirements) into Software Engineering is often referred to as DevOps.

We recommend introducing a top-level Integration Stream to pull together Releases for transition (performing top level Quality Assurance activities) to live environments either manually or automatically using tools such as Puppet or Chef.

By separating levels of Done with Integration Streams, the lowest streams are safe for collaboration and bug fixing whereas higher level streams only have higher quality changes promoted to them. In turn their higher level of quality makes them good candidates for safely pushing to live environments.

This approach gives teams the best of both Continuous Integration and Continuous Deployment without exposing every single change to live environments.

The term “DevOps” refers to tight integration between the Software Development and Operations parts of an organization.