Clogging the Continuous Delivery Pipeline: A Look at Top Bottlenecks

Continuous Delivery is an appealing approach, allowing rapid application updates, so end users get features and fixes as soon as they’re ready. Developers, for their part, can fix and push updates to address bug reports almost as quickly as they come in. However, the process is not foolproof. A recent Enterprise Management Associates (EMA) survey of IT companies using the Continuous Delivery method identified nine points that often bottleneck the CD pipeline.

Continuous delivery

1. Manual Troubleshooting Processes Dealing with Production Environments

Troubleshooting live code in production is immensely disruptive to the CD process. Sometimes bad code gets through testing servers, or end users find ways to use the platform that developers never intended or foresaw. In these stressful scenarios, end users are directly interacting with the problems, meaning dev teams have to re-prioritize to fix live problems before working on anything else. There isn’t much a team can do to improve manual troubleshooting in the production environment, so prevention is always the best approach. Utilizing and improving testing automation before pushing code live can help identify and debug problems beforehand.

2. Time Spent Testing Code

Developers know that applying fixes and updates to code is only the beginning of the process. Devs spend just 30 percent of their time writing the code, and 70 percent of their time reading and testing it. Without the proper debugging tools and automation processes, the amount of time spent on testing can spiral out of control. Many dev teams have found it helpful to predetermine how much time they will spend testing code to avoid this common problem. Moreover, team confidence in the Continuous Integration server can help with pushing code into production faster.

3. Manual Processes Supporting Package Builds and Production Deployment

Teams report that pushing package builds and deploying updates are major delays. Developers and testers are so busy making changes to the application and making sure everything works that they aren’t taking the time to push updates as soon as they’re ready. These steps are not automatic and require a gatekeeper to apply changes.

4. Fear/Risk Related to Adverse Production Impacts of New Code

At first, developers are deliberate and methodical when debugging and testing their code. As they become more confident in their abilities, however, they spend less and less time testing until they screw up and release an unprepared update that leads to bottleneck number one. These situations create a rational fear of releasing bad code, but the experience can lead to an irrational response where teams refuse to release code without excessive, unnecessarily complex testing. This is as much an organizational issue as it is an infrastructure issue. The dev team needs to be confident in its programming skills, testing practices, and CI testing server to alleviate any fears.

5. Time Spent Creating Code

Writing the code can be a major bottleneck. This holdup typically stems from a combination of programming inexperience and platform unfamiliarity. It’s common for developers to work on multiple projects that require different programming skills. Problems also stem from poorly organized and documented application code. The best ways to resolve delays in the code creation process include training devs in the necessary languages and platform operations, as well as intelligently organizing, separating, and documenting code.

6. Manual Processes Supporting Code Movement Through Life Cycle Stages

Updates can fall behind if they don’t make it through the programming life cycle stages in a smooth manner. For example, when a code update gets caught in a loop between the analysis and design stages or the implementation and testing stages, it can hold up additional updates that are functioning correctly from making it through the process.

7. Lack of Testing/Quality Assurance Personnel

Teams also reported that staffing shortages in the testing and QA departments often hold up the process. In many organizations, the same people who write the code are also the ones who do the testing and QA work. Staffing shortages disrupt the assembly line flow of the CD process.

8. Lack of Testing Facilities, Developers, and QA Needing to Wait

The CI and CD processes work best when the development team is using a branching pattern to break off parts of the applications into smaller, more manageable chunks. Failing to do this can prevent QA staff from being able to test code because devs are holding back a feature update while working on other code. Updates can also be delayed because the testing infrastructure is unable to keep up.

9. Lack of Workflow/Automation Tools to Drive the Continuous Delivery Pipeline

Development teams report problems with workflow and automation tools delaying CD. The source of this issue is often a lack of high-quality tools available to help manage testing for constant updates.

For more information on how Apica’s stress testing and test automation solutions  for high-performance applications can automate time-consuming parts of the development process, check out this white paper.

or see a demo of Continuous integration with AWS Codepipline.