Month: March 2020

Article originally posted on InfoQ. Visit InfoQ

Recently Amazon announced Global Datastore, a new feature of Amazon ElastiCache for Redis that provides fully managed, fast, reliable and secure cross-region replication. 

With the Global Datastore, Amazon delivers a feature to its ElastiCache for Redis following customer demand for replication of clusters across AWS regions. Writing data into ElastiCache for a Redis cluster in one region leads to the data available for reading in two other cross-region replica clusters when creating a Global Datastore – thus enabling low-latency reads and disaster recovery across regions. 

Users can setup a Global Datastore by starting with an existing cluster, or by creating a new cluster to be used as a primary using either the AWS Management Console for ElastiCache – or through automation by downloading the latest AWS SDK or CLI. By using the AWS Console, users can create the Global Datastore specifying an existing primary cluster or creating a new one.

Source: https://aws.amazon.com/blogs/aws/now-available-amazon-elasticache-global-datastore-for-redis/

Next, the user fills in the details for the secondary cluster hosted in another region, and potentially adds another cluster if necessary. Note that users can add up to two secondary clusters in other regions which will receive updates from the primary.

Julien Simon, artificial intelligence & machine learning evangelist for EMEA, stated in the blog post about the Global Store feature:

Last but not least, clusters that are part of a global datastore can be modified and resized as usual (adding or removing nodes, changing node type, adding or removing shards, adding or removing replica nodes).

OpsGuru, an AWS and Google Cloud partner which provides consulting, implementation and managed services for a wide range of cloud-native workloads, said in a tweet:

Though #Redis has been introduced as a temporary cache, many use cases use Redis as a key-value data store; that’s why this new feature of multi-region replication is so vital for a Redis-dependent workflow to be truly resilient across regions.

Other major cloud vendors also provide disaster recovery capabilities with their caching services. Google, for instance, offers a caching service with Memorystore, which on the standard tier supports replication in a region for failover. And Microsoft has Azure Redis Cache, which supports clustering on their premium tier.

The new Global Datastore feature is available with Amazon ElastiCache for Redis 5.0.6 or above and supported on M5 and R5 nodes. Furthermore, the Global Datastore feature in ElastiCache for Redis is available in various AWS regions in the US, Asia Pacific, and Europe at no additional cost. Note that Amazon ElastiCache node-based pricing is applicable for all nodes created as a part of Global Datastore and standard AWS data transfer out rates apply. More details on pricing are available on the pricing page of Amazon ElastiCache for Redis.

Article originally posted on InfoQ. Visit InfoQ

Google recently presented MediaPipe graphs for browsers, enabled by WebAssembly and accelerated by the XNNPack ML Inference Library. As previously demonstrated on mobile (Android, iOS), MediaPipe graphs allow developers to build and run machine-learning (ML) pipelines, to achieve complex tasks.

MediaPipe graphs are visualized in a browser environment with MediaPipe Visualizer, a dedicated web application that allows developers to build a graph consisting of nodes representing different machine learning or other processing tasks. The following figure corresponds to the running of the MediaPipe face detection example in the visualizer.

Source: https://developers.googleblog.com/2020/01/mediapipe-on-web.html

As is apparent from the graph, the face detection application transforms input frames (input_frames_gpu) into output frames (output_frames_gpu) through a series of transformations including the conversion of incoming frames into image tensors (TfLiteTensor), posterior processing by a TFLite model for face detection, and overlaying of annotations on the output video.

The visualized graph matches the facing text which contains a description of the nodes’ content and the expected processing to realize. The MediaPipe Visualizer will react in real-time to changes made within the editor in order to maintain the correspondence between text and graph. The configuration of the previous TFLite model is for instance as follows:

# Converts the transformed input image on GPU into an image tensor stored as a
# TfLiteTensor.
node  {
calculator:  "TfLiteConverterCalculator"
input_stream:  "IMAGE:transformed_input_video_cpu"
output_stream:  "TENSORS:image_tensor"
}

Google created four demos (Edge Detection, Face Detection, Hair Segmentation, Hand Tracking) to be run in the browser.

The browser-enabled version of MediaPipe graphs is implemented by compiling the C++ source code to WebAssembly using Emscripten, and creating an API for all necessary communications back and forth between JavaScript and C++. Required demo assets (ML models and auxiliary text/data files) are packaged as individual binary data packages, to be loaded at runtime.

To optimize for performance, MediaPipe’s browser version leverages the GPU for image operations whenever possible, and resort to the lightest (yet accurate) available ML models. The XNNPack ML Inference Library is additionally used in connection with the TensorflowLite inference calculator (TfLiteInferenceCalculator), resulting in an estimated 2-3x speed gain in most of applications.

Google plans to improve MediaPipe’s browser version and give developers more control over template graphs and assets used in the MediaPipe model files. Developers are invited to follow the Google Developer twitter account.

MediaPipe is a cross-platform framework for mobile devices, workstations and servers, and supports GPU acceleration. MediaPipe is available under the MIT open source license. Contributions and feedback are welcome and may be provided via the GitHub project.

Article originally posted on InfoQ. Visit InfoQ

Introduced in Git 2.18, Git wire protocol version 2 is now used by default in Git latest version, 2.26. Git 2.26 also improves configuration option handling and sparse-checkouts, among other things.

Git’s wire protocol defines how Git clients and servers communicate with each other. The new Git protocol version improves performance by enabling server-side filtering of references, which include not only branches and tags but also, e.g., pull request heads and others. Clients adopting Git protocol version 2 will be able to specify which references they are interested in, thus reducing the amount of data the server sends back. Contrast this with Git original protocol behaviour, where the server starts by sending back a list of all references in the repository, which could amount to many megabytes.

According to figures provided by Google engineer Brandon Williams at the time of Git protocol version 2 announcement, the new protocol can be significantly faster than the old one, especially with large repositories such as Chrome’s, which contains more than 500k references. Additionally, noted Williams,

Protocol v2 has also enabled a reduction of 8x of the overhead bytes (non-packfile) sent from googlesource.com servers. A majority of this improvement is due to filtering references advertised by the server to the refs the client has expressed interest in.

The reason for the delay in making Git protocol version 2 the default, notes Taylor Blau on GitHub blog, has been giving enough time for developers to catch any bugs in the protocol implementation. Interestingly, Git protocol version 2 is designed so any client implementing it can still talk to a Git server only supporting the old Git protocol.

As usual for Git releases, Git 2.26 includes a very long list of performance improvements, fixes, and new features. For example, git clone --recurse-submodules --single-branch now applies the --single-branch option to submodules as well.

Additionally, the git sparse-checkout command, introduced in Git 2.25, has got a new add subcommand which can be used to add new directories to your sparse checkout one at a time. Sparse checkouts are mostly useful with large repositories, where you are not interested in the whole repo content but only on some of its subdirectories. In Git 2.25, to add a new directory to those already included in your sparse checkout, you were required to list all of them when using the set subcommand.

Worth of mention is also an improvement to the diff family of commands, which enables git add -p to better deal with whitespace problems.

As a final note, Git 2.26 makes it easier for you to know where one default option is defined, whether at repository, user, or system level, using git config --show-origin.

Do not miss Git 2.26 official release notes if you are interested in the full detail about this release.

Article originally posted on InfoQ. Visit InfoQ

Anton Tarasiuk discusses the legal issues that can be encountered when dealing with Big Data, GDPR and contracts.

By Anton Tarasiuk

Article originally posted on InfoQ. Visit InfoQ

Rod Cope discusses a continuous testing approach that fits into the DevOps pipeline, providing a formula for mobile app testing.

By Rod Cope

Article originally posted on InfoQ. Visit InfoQ

Johnny Hooyberghs introduces Quantum Computing, explaining why it matters, what is a qubit and superposition, what is IBM and Microsoft doing in this area, and what are some of the algorithms used.

By Johnny Hooyberghs

Article originally posted on InfoQ. Visit InfoQ

Subscribe on:

Apple Podcasts Google Podcasts Soundcloud Spotify Overcast RSS Feed

. From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.

Article originally posted on InfoQ. Visit InfoQ

The DevOps movement continues to grow in terms of its impact and influence in the IT ecosystem. There’s no question that the DevOps movement is maturing to the point that it has a power and a voice comparable to that of other movements such as cloud and agile. One interesting characteristic of the DevOps movement is that it arose from a wish within IT to optimize and harmonize the activities of different teams; but it’s matured to take into consideration the needs of end users and a holistic view of how organizations need to operate in order to be both flexible and efficient at scale.

Business leaders may complain that IT workers don’t appreciate the underlying business context in which they’re operating; and historically the IT department has been seen as a cost centre and not a value centre for organisations. The net result of that is that IT struggled to justify infrastructure and process improvements that would increase costs, unless they were specifically tied to a new business-facing initiative. But IT is becoming increasingly central to business operations. Done well, the flexibility of software allows organisations to adapt much more quickly than in a pre-digital infrastructure, thus making effective IT teams the biggest enablers of value.

Project to Product

Digital transformation has become an incredibly popular buzzword in recent decades, but it is essential for companies to recognize that digital transformation is not a one-time activity. The work done by Mik Kersten in his book ‘Project to Product‘ highlights the fallacy of thinking of IT improvements as one-time projects. Digging into the history of technological revolutions and witnessing the most advanced manufacturing processes of today helped Kersten recognize that shifting to a product-focused mindset is a far more beneficial way for companies to frame IT.

What does it mean to approach internal IT initiatives as products? One aspect is recognising that IT systems entail long-term maintenance responsibilities, and are not something that should simply be created and turned over to an unrelated team. Another aspect is re-thinking the way we finance projects. The annual budgeting cycle is optimised for relatively stable systems, where change can be anticipated a year in advance. Teams vie for a share of the budget, and ensure that they never underspend their allocation, lest they receive less the following year. The requirement to get up-front funding tends to promote large, expensive projects, while budget for subsequent operation is often not factored in. The result of this are complex IT systems that are maintained by teams other than the ones who built them, that are excessively complex, and that often don’t meet the needs of end users. Agility needs to extend into every aspect of a business, from how teams are structured, to how work is financed, to how and when an organization decides to make improvements.

An example of a product-mindset is to have the team in charge of a CPQ (Configure – Price – Quote) system be responsible for both building and maintaining that system over time – the “you build it, you own it” mantra. The power of this mindset is in promoting stability and flexibility in what is built. When a team has long-term responsibility for building and maintaining the product, and for getting feedback on how it’s used in production, they greatly increase the chances of this product bringing lasting value to the organisation. Feedback on how systems are used in production is the essential input for ongoing improvement. It’s for this reason that in the DevOps community the assembly line motif of software development has given way to a circular or infinity loop motif for software development. The loop indicates that ongoing feedback from end-users is as important as the original specifications.

Product companies and product teams are by nature more stable than teams who just build and move on. Project teams work great in construction and civil engineering, where infrastructure rarely needs to change once built. But software allows for an utterly different level of agility. It can be replatformed or refactored without end users being aware; its design or functionality can be changed at the last minute before release to users; and different versions can be exposed to different users to solicit feedback. This agility is why the knowledge that went into creating a software product must be retained in the team if at all possible. There is a “cone of uncertainty” where at the outset of building something we know the minimum that we will ever know about it. As we proceed with development and a system comes under use, we necessarily learn things that could never have been predicted even by the most intelligent and thoughtful planning team.

Team Topologies

Related to this topic of “project to product” is the research that has been done around team cognitive load, and the importance of structuring teams in a way that optimises for communication and trust. There are both mathematical and empirical bases for saying that smaller teams that can act independently will perform more effectively than large teams or teams with many dependencies. Amazon’s two-pizza team size was not created to simplify their catering needs. It’s a wise structure that maximizes the power of teamwork while minimising the overhead required for constant coordination. Conway’s Law dictates that your architecture will come to reflect your team structure. Given that an optimal team size is constrained to five to ten people, modular architecture is essential if you want to be able to maintain your velocity over time. The idea is to give teams relative autonomy over particular systems. This feeds into topics like continuous integration which strives to reduce the impact of parallelisation. These kinds of DevOps best practices began with research and experimentation in the field, moved to being DevOps community lore, became substantiated through the research of teams like DevOps Research and Assessment, and will gradually become the basis for dedicated tools and entrenched ways of working.

Much of the tool growth that we are seeing today includes integration and Value Stream Management platforms such as Copado, Tasktop, Plutora, Xebialabs, GitLab and Cloudbees who are striving to bring information from many disparate systems together into one tool. I expect this integration and aggregation trend to continue as a practical way of dealing with the diverse reality of enterprise systems. In fact, teams benefit greatly from being able to choose their own tools. In the book ‘Team Topologies’ by Matthew Skelton and Manuel Pais, they refer to standardization as one type of monolith, “monolithic thinking”, that can interfere with maximum effectiveness. If you are striving to avoid monolithic thinking, but nevertheless need an integrated view of your systems and processes, data integrations are your only option.

Tool Evolution

LaunchDarkly showed that there is a market for products that facilitate particular DevOps practices, in this case separating deployments from releases. That practice is integral to activities like A/B testing and canary deployments, which have become recognised as powerful ways to reduce risk and enable experimentation. I would expect that more tools continue to appear to enable DevOps practices that otherwise would require custom coding.

Although this is not a new trend, definitely expect to see services like Amazon Web Services (AWS), Azure and Google Cloud Platform (GCP) continue to roll out new service offerings, compete with one another, and gain market share from legacy infrastructure. Azure and GCP are still playing catch-up in terms of the variety of products that they offer as well as the market share that they control, but expect them to be following close on the heels of AWS. Offerings such as kubernetes-as-a-service help to reduce the complexity of managing underlying infrastructure. Expect other kinds of complex systems to continue to be bundled into turnkey applications.

Data Science

The State of DevOps Reports from Puppet and Google have set the standard for using data science to evaluate the effectiveness of development practices. Expect to see more tools begin to integrate analytics and data science into their offerings. And expect to see more teams requesting and making use of such capabilities to facilitate experimentation and to validate the results of those experiments.

Business functions such as marketing have been using A/B testing and quantifying their effectiveness for many years. There are a huge number of marketing-oriented tools that have been highly tuned to give metrics on click through rate, adoption, time on site, return on investment, etc. The most long-standing of these is Google Analytics, but marketers have a vast range of tools to choose from. Ironically, IT teams are late to the party in terms of practices such as A/B testing and ensuring that applications are being adopted by users. It is often left to business teams to track adoption of an application created by an IT department. But internal IT departments are the ones who have the greatest opportunity to make improvements to meet the needs of users.

Expect to see tools for usage monitoring and even embedded feedback and user satisfaction surveys becoming more frequently used by internal development teams. These practices help close the gap between end users and development teams in the same way that marketing teams have been striving to close the gap between companies and their customers for many years. This kind of feedback is exceptionally powerful since there’s an inefficiency introduced by involving a business analyst every time you want to solicit user feedback. Selective feedback initiatives also suffer from sample bias.

Lean and Agile

DevOps is aimed at “actualizing agile” by ensuring that teams have the technical capabilities to be truly agile, beyond just shortening their planning and work cadence. Importantly, DevOps also has Lean as part of its pedigree. This means that there is a focus on end-to-end lifecycle, flow optimisation, and thinking of improvement in terms of removing waste as opposed to just adding capacity. There are a huge number of organisations and teams that are still just taking their first steps in this process. For them, although the terminology and concepts may seem overwhelming at first, they benefit from a wide range of well-developed options to suit their development lifecycle needs. I anticipate that many software tools will be optimizing for ease-of-use, and continuing to compete on usability and the appearance of the UI.

Whereas most early DevOps initiatives were strictly script and configuration file based, more recent offerings help to visualise processes and dependencies in a way that is easily digested by a broader segment of the organization. Especially as companies try to capture the attention and wallet share of CIOs, CTOs and other organizational decision-makers, reducing the difference between the actual UI of a tool and how it can be visualized and marketed becomes even more important. The most beautiful and graphical tools just sell themselves (to both business and technical users). Value Stream Mapping and delivery pipelines are particularly popular and effective ways to visualize the delivery process, while also providing day-to-day access to metrics and monitoring.

System Integrators

Finally, it’s clear that the market and demand for practices such as DevOps exceeds the availability of skilled people to implement it. Thus system integrators will continue to have a role in helping teams to ramp up on these technologies and processes, and in some cases to even help manage the development lifecycle for organizations. Research in the State of DevOps Reports indicates that functional outsourcing is predictive of low performance, so organizations should be very careful about delegating one particular activity such as testing or deployment to an outsourced contractor. Unless done carefully, functional outsourcing goes against the spirit of DevOps which focuses on bringing all of the relevant stakeholders (from Dev to Ops) together with aligned goals, shared visibility, and shared technologies.

Consulting partners are an extremely powerful way of getting help in the absence of in-house talent. But they necessarily introduce organisational boundaries unless consultants are deeply embedded in the organisation, working long-term alongside full-time employees. Rely on DevOps consultants as enablers to help you to adopt technologies, improve your processes, design your metrics and so forth. But be careful about outsourcing a particular part of your process (such as testing) that is on the critical path to production.

It may work better to give the entire development process for a particular application to a consulting company. But consider the long-term lifecycle of this and how you want that company to maintain the application over time. In the spirit of project to product, there is risk in having one team build an application and a separate team maintain it. Think of the knowledge that you are building within the team as a critical part of your architecture. Just as it’s foolhardy to rip out a substantial chunk of your architecture just after go-live, so too it’s unwise to rip out a substantial chunk of knowledge from your team just after go-live.

Summary

In summary, the DevOps movement continues to grow, flourish, and gain influence in the IT world and the business world at large. As our organisations become increasingly digital, the agility of our IT systems becomes critical to the life and health of our companies. DevOps as a movement blends together psychology, sociology, technical management, automation, security, and practices such as lean and agile to optimise an organisation’s ability to thrive in a digital world. The consulting, tooling, infrastructure, and training ecosystem to support this is still evolving. The market for DevOps is in fact the market for digital success, thus expect continued growth through 2020 and beyond.

About the Author

Andrew Davis is a Salesforce DevOps specialist who’s passionate about helping teams deliver innovation, build trust, and improve their performance. He currently works as Senior Director of Product Marketing for Copado, helping people understand the importance of DevOps for scaling Salesforce implementations.

Article originally posted on InfoQ. Visit InfoQ

Rome is an experimental JavaScript toolchain created by Babel and yarn creator Sebastian McKenzie and the React Native team at Facebook. Rome includes a compiler, linter, formatter, bundler, and testing framework, aiming to be “a comprehensive tool for anything related to the processing of JavaScript source code.”

Unlike similar projects which combine various tools into a workflow, Rome is a rethink of the JavaScript ecosystem with a bold approach to the entire toolchain with no pre-existing third-party dependencies.

Rome is authored in TypeScript and is a monorepo with internal packages to delineate code boundaries. Rome is self-hosted and compiles itself with an old version of Rome. Rome provides support for processing JSX, Flow, and TypeScript code.

The Rome Getting Started documentation is currently intentionally very sparse, with information about initializing Rome, defining settings in a JSON file, and actions including running, linting, compiling and parsing source code.

Preact and Jason Miller created an overview of Rome and detail usage of the available CLI commands, compare its bundling output to Rollup, provide a few examples, and other early insights.

The Rome project has a clear set of development guidelines, from clear and easy to understand error messages, as small of an API footprint as possible, and strongly typed code. The Rome Team is currently focused on linting and maintain a list of actively considered issues. As explained by McKenzie:

The current area of focus for Rome is linting. This is the path of least resistance for finishing a part of Rome that can be used most easily.

Rome offers a promising rethink to perhaps remove cruft from the current state of JavaScript development and tooling, though it is far too early to determine if this approach will change the status quo or simply lead to another option for the JavaScript toolchain. Community interest is strong and includes discussion on whether Rome might include future support for WebAssembly.

Rome is available under the MIT license though it is not ready for production use and currently can only get used by building from source. Rome is open for contributors interested in experimental tools following the contribution guidelines and code of conduct.

Article originally posted on InfoQ. Visit InfoQ

InfoQ Homepage Presentations Playful Leadership: The Art and Science of Emotions

• Related Editorial