Article originally posted on InfoQ. Visit InfoQ

The software delivery process has been transformed in the last decade; we’ve adopted  well-understood workflows around functions such as testing, release management and operational support.

These changes have enabled organisations to speed up time to market and improve service reliability. So why does security still seem so far behind? And why does it still feel so hard for so many product teams? 

In this article we’ll explore the impact that typical security workflows have on software delivery, explain the root causes and share battle-proven techniques through case studies to show how we can make delivering secure software easier.

A typical security workflow

First of all, let’s look at a typical secure delivery workflow and identify some common issues.  If you’re nodding your head along to these, then this article is for you.

Workflow purpose

Most organisations have risk assessment and security programs that directly relate to regulatory standards; they’ve been designed to ensure that all software is subject to industry-standard security checks, ensure that any risks that end up in production are classified, and that somebody is held accountable.

This is all good stuff which most people would agree with, however in practice we often see security workflows that were designed to service waterfall-based projects with infrequent bing-bang releases, which haven’t evolved to support modern software delivery and make regularly releasing small chunks of software difficult.

What happens in practice

Using these workflows, a product delivery team will often develop a system without technical security standards or requirements, then when they’re just about ready to release they begin to review their system architecture with a risk assessor who tries to spot any security concerns in the design and recommends more effective security controls.

When reviewing architecture at a high level, any security concerns are usually big-ticket items that require considerable effort to retrofit, and sometimes even the redesign of a critical feature of a system like authentication.

Lower-level threats and vulnerabilities are often found by outsourcing deeper technical security knowledge from an external penetration testing company, who are engaged to attack the system and highlight any serious issues.

After these activities are complete, we usually see a fractious negotiation around risk and resources, with the engineering team pushing back on making expensive, time-consuming changes to their system architecture and operational processes just before their release deadlines, and the system owner pushing for risk acceptance for all but the most serious risks.  

Overall, security can be seen as something that’s owned by the security team and not an attribute of a system’s quality that’s owned by engineers, like performance or reliability.  The result is often a system which has sub-optimal security controls and a growing risk register of known issues which are rarely re-examined.

What is the impact?

When teams use the workflows detailed above, we commonly observe the following negative impacts:

• Security activities slow down delivery and reduce release cadence – Time-consuming workflows to initiate and perform big bang security reviews encourage larger, less frequent and higher risk software releases.
   
• Improving security involves lots of duplicated effort across many teams – Each team performing the same assessments or penetration tests on every release which produce similar findings.
   
• Large chunks of security work just before release – Sudden demand for large architectural system changes to implement new security controls based on an end-of-project risk assessment.
   

So now that the problems are clear, how can we improve? 

Break up the big bang security review and do the right activities at the right time

Delivery teams can proactively use their skills and agile experience to break up the large, waterfall security risk assessments into more manageable, granular security review activities that can fit modern workflows and rapidly changing systems.

Case study

While working with a regulated financial services institution on a new customer-facing product, each software release was subject to a full risk assessment and penetration test which forced big-bang releases with a cadence of around three months, which severely impacted the quality of the product as releases were full of bugs and user feedback was incredibly slow.

Categorising releases

We first worked with InfoSec teams to categorise different types of releases, based on the scope of architectural changes, whether they related to security sensitive controls like authentication, whether the change introduced new classifications of data with different protection requirements, or whether they were small incremental changes to functionality.

Sensible review activities at the right time

Once we agreed on these release categories, we were able to introduce a process where each user story was self-assessed by a security champion at the backlog refinement sessions (with the result added to the JIRA ticket) and based on the agreed release criteria we established the type of security activities that were required.

The range of activities were Peer Code Review, Static Code Scan, and Threat model, Risk Assessment / Penetration test.

Provide assurance

Thanks to linking evidence of each self-assessment decision to each user story, and providing links to security scans, pull request reviews or threat models, delivery teams could use Jira Query Language to provide an instant report that demonstrated all releases had undergone the right levels of security review, and when audit time came along it was easy to show specific examples for any chosen release.

Reap the benefits

By self-assessing each user story and performing the most applicable security activities for each release based on agreed criteria, we were able to realise the following benefits: 

• Increase our release cadence to weekly releases, delivering value faster and reducing the number of bugs
• Avoid expensive retrofitting work near the end of a project
• Improve the security of our products as each small change had the benefit of security input while in development, rather than as a big bang at the end of three months of work

Scale out common and expensive security controls

Product delivery teams can pre-define common security requirements, develop approved patterns and build shared security capabilities, allowing them to easily design effective security controls into their systems (or get them for free).

Case study

While working with a PCI-regulated organisation during a transformational period, several product delivery teams were actively building brand-new products using a microservices architecture.

Mandatory architectural security reviews were carried out once development was complete, with important security requirements around areas like data protection, authentication and authorisation, and secrets management frequently surfacing once systems were functionally ready for release.  Each team would build their own implementations of these controls and re-submit them for review, causing extensive project delays and large amounts of duplicated work.

Identify common security-critical components

We reviewed all previous security assessments, finding the common areas which required intensive security review and led to expensive retrofitted security controls.  The areas we noticed were causing repeated problems for multiple teams were:

• Secrets Management
• Application Security Logging
• Client Authentication / Authorization

Choose a practical method to standardise and scale

Depending on how a product delivery function is structured, the technology they use and their architectural ethos, several different techniques can be used to help standardise security controls and make them cheaper for teams to implement early:

• Pre-agreed technical security requirements
• Shared code libraries or infrastructure-as-code modules
• Fully supported security cross-cutting capabilities

Technical security requirements

Many new products implemented different authentication and authorization schemes based on the needs of various backend third-party integrations.  Technical security input was not available throughout product delivery, and any security issues were usually identified after pre-go-live risk assessment or third party penetration test.

The product delivery teams reviewed all previous penetration test findings and worked with a security subject matter expert to work on defining a set of core requirements that teams could use when designing their authentication schemes.  We built out technical requirements documents around areas like approved crypto algorithms, password hashing strategies, rate limiting and key rotation, and these standards were used during the design phase for all further projects, saving large amounts of retro-fitting and re-design.

Shared code libraries

We realised that each delivery team was subject to a drawn-out process of identifying security-relevant events in their applications, implementing specialised logging to capture these events and then working with security teams to test their effectiveness in a centralised SIEM (Security Information and Event Management) tool.

We worked with security teams to define a general set of useful security events (For example: Failed login, Rate limit reached, Signature verification failure), and implemented a shared code library which teams could import into their applications that provided a standardised way of logging these events which the SIEM understood.  We released the library with easy-to-understand documentation that allowed teams to build in the required logging while building their applications, and saved weeks of project delay in many projects.

Cross-cutting capabilities

One of the largest areas of risk present in the new microservices was around secrets management.  Many delivery teams had developed their own secrets management mechanisms, ranging from encrypted values in source control to basic encrypted cloud object storage.  The lack of standardisation meant each team spent weeks working back-and-forth with InfoSec teams to reach a security compromise, and responding to any incidents regarding secrets was disjointed and often required key individuals’ knowledge.

We implemented a generic and secure secrets management mechanism which could be used by any deployment, so teams could inject secrets using placeholder values in config files backed with a secure cloud-based secrets storage mechanism.  Protecting secrets was much easier given the standardised cloud logging and access management capabilities, teams spent almost zero time building secrets management into new products and demonstrating that secrets were securely managed for audit and compliance reports was incredibly easy.

Conclusion

If waterfall-based security workflows are slowing your delivery down, or you realise that large chunks of your valuable development time is spent retrofitting security controls, you can treat these issues as just another value stream problem to solve and invest in using these techniques to develop the workflows, knowledge and capabilities required to build software securely and at pace.