Article originally posted on InfoQ. Visit InfoQ

This article discusses real-world cases of using shared libraries, their consequences, and possible solutions to blockers caused by using them in many dependent projects.

The challenges and solutions presented here are focused on .NET projects, but the suggested solutions could be adapted to other technologies and languages.

It also seeks to encourage the thoughts of software architects and developers to analyze trade-offs before creating and using shared libraries.

Understanding Shared Libraries

Shared libraries are reusable bundles of code designed to perform common tasks across multiple projects. They save time, ensure consistency, and prevent developers from having to reinvent the wheel.

These libraries are also known as client libraries or packages. They can be internal or public, versioned (enabling developers to control changes), and require ways to manage and distribute them across different projects.

Best Use Cases for Shared Libraries

The primary motivation for using shared libraries is to standardize and simplify development processes. However, there are trade-offs that we must be aware of when deciding to create and use a shared library, as Ben Morris presents, and we summarize here:

Some expected benefits usually considered are:

• Efficiency improvement and code quality
• Prevent duplicate work and standardize solutions
• Better modularity and abstraction
• Improved collaboration

But shared libraries often create unexpected problems:

• Introduce coupling between teams
• Cause version compatibility challenges
• Risk of potential breaking changes or regressions

Shared Library Management

Once an organization decides to adopt shared libraries as part of its architecture strategy, the next step is to establish an effective way to distribute and manage these libraries across multiple microservices. This is typically achieved by hosting an internal artifact repository. Solutions like AWS CodeArtifact, JFrog Artifactory, or equivalents are commonly used for this purpose. These systems allow teams to publish and consume packages (such as NuGet for .NET), ensuring shared components are accessible and version-controlled.

In the .NET ecosystem specifically, NuGet packages are the standard for managing and distributing shared libraries, making it easy for teams to integrate common functionality across services. However, while maintaining and publishing updates to the shared library itself is a manageable task, the real challenge lies in ensuring that all dependent services consistently adopt these updates.

As the system landscape expands, outdated dependencies scattered across services can quickly evolve from a minor inconvenience into a critical operational risk. These stale versions may lead to inconsistencies in behavior, expose security vulnerabilities, and cause integration issues between services. If left unmanaged, the situation becomes a scaling bottleneck, complicating deployments and slowing down the overall delivery pipeline.

Real-World Challenges

To understand the complexities of managing shared library versions in different projects, let’s analyze the following real-world cases involving internal shared libraries in .NET projects.

Real-World Scenario: Managing Authentication in AWS Lambda

• Scenario: The team developed a shared library to handle authentication logic for 10 Lambda functions triggered by AWS API Gateway endpoints. These functions were built using .NET 8 and used the shared library to verify whether the customer and user requesting data were trustworthy. The decision to use a shared library was made to consolidate common functionality and reduce code duplication. However, the library evolved to include not only authentication classes but also API content-negotiation classes and error-handling logic, which introduced additional dependencies across multiple services.
• Challenge: A new authentication parameter was introduced during development, requiring updates to all dependent Lambda functions before deployment.
• Impact: This dependency delayed the rollout of the new functionality, highlighting the risk of tightly coupled services. If it were in production, this would not happen without breaking some external APIs already consuming those endpoints.

RabbitMQ Integration: A Scalability Challenge

• Scenario: A specific internal shared library was built to handle connections and queues with RabbitMQ in this case. About 50 microservices run under a Kubernetes cluster and must integrate with RabbitMQ. The same team manages all microservices. The background for this decision is a small company that rewrote its system from a monolith, dividing it into many microservices, but with few changes after they went into production.
• Challenge: RabbitMQ must be upgraded from version 3.13.7 to 4.0.4 to ensure ongoing support and access to newer queueing models, notably quorum queues. However, this upgrade introduces a breaking change: queue declarations now require a new parameter, incompatible with the current shared library and all existing microservice implementations.
• Impact: There is a deadlock situation: If RabbitMQ is upgraded before the microservices, all services will fail when attempting to connect or create queues. If microservices are upgraded before RabbitMQ, they will attempt to use quorum queues not yet supported by the current RabbitMQ instance, breaking message flow immediately. This creates a zero-tolerance migration window, requiring all services to be upgraded in sync. This is a highly complex and risky operation given current resource constraints. The upgrade to RabbitMQ 4.0.4 has been postponed until the team finds the best way to migrate all microservices simultaneously. Without a clear and executable migration strategy, the team remains blocked from adopting critical RabbitMQ improvements. This increases technical debt and risks a catastrophic outage in the event of an uncoordinated upgrade or library change.

Insights: Dependency on shared libraries can become a critical blocker in scalability scenarios.

Keeping Services Synchronized with Shared Library Changes

1. Manual Updates: A Traditional Approach

The first method that comes to our minds to handle all dependent service updates is to manage them manually. Each service should have a team responsible for maintaining and incorporating it into the development process to keep it up to date.

However, this method may take a long time until all services are updated, as all teams must be aligned, and the tasks must be prioritized in each team’s backlog. It can also be time-consuming and error-prone.

2. Centralized Dependency Version Management

An alternative to the manual approach is to manage the shared libraries that the services depend on in a centralized way, considering external files or tools/features that handle that. Tools like this and established processes that include Continuous Integration/Continuous Deployment (CI/CD) pipelines with regression tests and automated deployment help achieve these expectations.

When updating a shared library, there is always a risk that the new version introduces breaking changes or unexpected behavior. Regression tests ensure that existing functionality remains intact after updates, reducing the chances of introducing new bugs when integrating a new library version.

Managing dependencies dynamically means frequent updates to services that rely on shared libraries. Automated deployment pipelines ensure that these updates are efficient, consistent, and error-free, minimizing manual intervention and reducing deployment risks.

CI/CD pipelines with regression tests and automated deployment are essential to enable centralized dependency version control. They help to validate dependency updates before they reach production. They automate tasks such as:

• Running regression tests
• Checking for compatibility issues
• Deploying new versions with minimal downtime

The idea is to accelerate the process of updating the dependent projects and enabling new features or tool upgrades while scaling and keeping a fast pace.

Development teams usually decide about structure at the beginning of their projects: either they will use a single repository for their code (known as a mono repo), have a solution containing one or multiple projects, but all kept in that same repository, or they will use multiple repositories to keep the code, creating different solutions with related projects, each solution divided by domain/responsibility.

For any of these decisions, we can manage dependencies using a .NET feature called Central Package Management, which is a feature from .NET 6+ that helps manage the dependencies for all projects in a solution. It makes it easier to control libraries and versions from a single place. We’ll refer to this as CPM for the rest of the article.

3. Strategies Based on Repository Structure

Mono-repo Strategy with CPM

When using CPM, we can manage all package versions from a single file (Directory.Package.props), which must be located in the same folder as the solution file (.sln). This .props file contains a list of all packages (with name and version), and each project file references the package by name that needs to be included in that project. Also, if one of the projects needs to use a version different from the one defined in the .props file, it can override that version specifically.

As we can see, CPM is primarily focused on a single solution containing multiple projects for managing dependencies. It simplifies package version management, as you can specify versions centrally in a single file. This improves solution performance, since all projects reference the same package version, reducing unnecessary restores and downloads. Additionally, it reduces maintenance effort, as updating a package version requires changing it only in a single file.

Multi-repo Strategy: CPM with Git Submodules

The real-world cases presented cover different areas and domains, but in both cases, each microservice has its own solution and respective git repository. When there are scenarios like the RabbitMQ Integration case, where multiple solutions consume the same internal shared library, a suggested approach to help manage these dependencies is to combine CPM with git submodules, leveraging one single central Directory.Package.props across all solutions.

As Git documentation explains, “Submodules allow you to keep a Git repository as a subdirectory of another Git repository. This lets you clone another repository into your project and keep your commits separate”.

For example, imagine you’re building a backend application that relies on a shared authentication module developed in a separate Git repository. Instead of copying the authentication code into your app, which makes it harder to track changes or get updates, you can add it as a Git submodule using git submodule add. This pulls the external repository into your project as a subdirectory, while keeping its commit history and versioning independent. You can then reference a specific version of the authentication module in your app, update it when needed, and maintain a clean separation between your core backend logic and the shared component, just like using a common internal library across multiple microservices.

The main idea here is that one repository stores only the dependencies prop file used by CPM (not a project or solution with code, just the dependencies prop file). In contrast, each service repository keeps only its code, so when we pull the repository, it works properly. We want to treat the two repositories as separate, yet still be able to use one from within the other.

The first step in implementing this idea is to create an empty repository containing only one file called Directory.Packages.props. This file will keep all packages and versions used by other solutions. The file content looks like this:

  
    true
  
  
    
    
    
  

Then, we can have one repository that contains the solution and projects with the code itself. In the example below, we can see the structure of a solution with two projects, sharedlibrarypost and sharedlibraryui. Each project contains its Directory.Packages.props too.

In these projects, the .props file only references the main Directory.Packages.props file (the one that contains the package names and versions). These .props files look like below:

In this solution, we can include the first repository as a git submodule through git commands. To achieve that, in the solution folder, we can use the following command:

git submodule add {.props github address} {folder}

An example would be: git submodule add github/example/centralizedpackages packagereferences where github/example/centralizedpackages is the .props github address and packagereferences is the folder.

After running that, the solution structure will look like below:

- /sharedlibrarypost
    ├── sharedlibrarypost.sln
    ├── packagereferences
    │   └── Directory.Packages.props
    ├── sharedlibrarypost
    │   ├── Directory.Packages.props
    │   ├── Program.cs
    │   ├── appsettings.json
    │   ├── sharedlibrarypost.csproj
    │   └── sharedlibrarypost.http
    └── sharedlibraryui
        ├── Directory.Packages.props
        ├── Pages
        ├── Program.cs
        ├── appsettings.json
        ├── sharedlibraryui.csproj

Once we have the structure ready as above, we must always run the following git commands to build the solution, and include the same commands as part of the build pipeline:
git submodule update –init –remote
dotnet build

To use this method, it’s recommended to have CI/CD pipelines with at least build, test, and publish stages. Tests are extremely important and must include unit tests, integration tests, and regression tests, fully covering the components. When a new version of the centralized Directory.Packages.props file is committed, this triggers the dependent projects pipeline, and the solution will pull the latest version to build. With the pipeline running, developers will be aware if an updated package version breaks any builds or tests.

An important caveat to consider when using submodules arises when working locally. The new versions of the Directory.Packages.props reflect automatically through the pipeline output for dependent services, but developers must explicitly run a submodule update to incorporate the latest version when working locally with the repositories. In practice, this can easily lead to situations where developers forget to pull and commit updated submodule references or continue working with outdated versions unknowingly. Such oversights might introduce inconsistencies and slow down development workflows.

4. Umbrella Package Strategy

An alternative approach to Central Package Management and Git submodules is the use of Umbrella Packages (also called meta-packages). This technique involves creating a dedicated NuGet package that groups and declares dependencies on other internal or external packages with specific versions.

By referencing this umbrella package, each project implicitly brings along all the necessary dependencies at predefined versions. This approach centralizes dependency control without needing to manage multiple configuration files or repositories.

For example, instead of having each microservice explicitly reference common packages like MyCompany.Logging, MyCompany.Security, or Newtonsoft.Json, they simply reference a single umbrella package, such as MyCompany.Platform.

The .nuspec file of this umbrella package defines dependencies as follows:

Once the umbrella package is published, let’s consider a microservice project that requires three shared libraries: MyCompany.Logging, MyCompany.Security, and Newtonsoft.Json. Below, we can see the content of the project file for each case:

Without the Umbrella Package:

With the Umbrella Package:

Using an umbrella package, the project only needs to reference a single package (MyCompany.Platform) that internally includes all required dependencies at the specified versions. This reduces clutter in the project file, promotes consistency, and simplifies version management.

When new versions of the dependencies are released, only the umbrella package needs to be updated and republished. However, it is important to highlight that consuming/dependent projects must still update the version of the umbrella package they reference in order to benefit from these updates. While this does not happen automatically, it simplifies maintenance since only the umbrella package version needs to be managed, rather than multiple individual package references.

⚠️ Note: Although the umbrella package centralizes dependency management, consumer solutions must manually update their reference version to receive the latest dependencies.

Advantages:

• Simplifies dependency management across multiple repositories.
• Reduces the risk of inconsistent versions between services.
• Keeps project files clean, as they reference only the umbrella package.
• Avoids the need to manage shared configuration files or submodules.
• Promotes standardization and internal dependency control.

Considerations:

• Requires maintaining and publishing the umbrella package regularly.
• Projects must update their references to the umbrella package version to receive updates to dependencies.
• Potential for version conflicts if consumer projects also reference packages independently.

5. Comparative Analysis

Trade-offs Between CPM and Umbrella Packages

Both Umbrella Packages and Central Package Management (CPM) aim to simplify dependency version control across multiple projects, but they differ in approach and suitability depending on the architecture.

Each approach offers distinct benefits and limitations that align with team workflows, repository structures, and release practices in different ways.

Explicitly outlining these trade-offs helps teams make informed architectural decisions based on their specific context, rather than adopting tools based solely on technical familiarity or convenience.

Use Umbrella Packages when working in multi-repository environments where minimizing configuration overhead in individual projects is essential. This approach is ideal for teams that prefer referencing a single meta-package that encapsulates all necessary dependencies, promoting consistency without managing multiple package references. It is particularly useful when repository independence is a priority and when teams are comfortable manually updating the umbrella package version to incorporate changes. Umbrella packages eliminate the need for shared configuration files or Git submodules, offering a clean and standardized way to manage dependencies across diverse services.

Use Central Package Management (CPM) in mono-repo setups or tightly coupled repository structures where fine-grained control over package versions is needed. CPM allows teams to manage all dependencies centrally through a Directory.Packages.props file, streamlining updates, and ensuring consistency across multiple projects within the same repository. This approach simplifies maintenance, as version updates only require a single change, and is well-suited for teams that prioritize strong alignment and efficient dependency management within a unified codebase.

Use CPM with Git Submodules when operating in multi-repo environments, but still seeking centralized control over package versions across repositories. By combining CPM with Git submodules, teams can share a centralized Directory.Packages.props file across multiple repositories, ensuring consistent dependency versions while maintaining repository autonomy. This method requires disciplined workflows to keep submodules updated both locally and in CI/CD pipelines, but it offers a balance between centralized version management and flexible, distributed development practices.

Conclusion

Shared libraries can either accelerate development or become a significant scalability bottleneck if not properly managed. To mitigate these risks, teams should adopt structured strategies tailored to their repository architecture.

Central Package Management (CPM) offers an easier approach for mono-repo setups, allowing centralized version control through a single configuration file. In multi-repository environments, integrating CPM with Git submodules provides centralized control while maintaining repository independence, though it demands disciplined workflows and consistent CI/CD pipeline support.

Alternatively, Umbrella packages offer a repository-agnostic solution, encapsulating all dependencies into a single package for simpler integration. However, they require manual updates to package references.

By leveraging these strategies with automated CI/CD pipelines and rigorous regression testing, teams can maintain system stability and scalability while efficiently managing shared dependencies.