Article originally posted on MongoDB. Visit MongoDB

NEW YORK, Aug. 30, 2019 /PRNewswire/ — MongoDB, Inc. (NASDAQ: MDB), the leading modern, general purpose database platform, today announced that it will present at two upcoming conferences: the Citi 2019 Global Technology Conference in New York City and the Deutsche Bank 2019 Technology Conference in Las Vegas.

• Chief Operating Officer and Chief Financial Officer, Michael Gordon will present at the Citi Conference on Friday, September 6, 2019 at 8:45 AM Eastern Time and will be webcast live.
• Mr. Gordon will also present at the Deutsche Bank Conference on Tuesday, September 10, 2019 at 7:25 AM Pacific Time (10:25 AM ET) and will be webcast live.

A live webcast of each presentation will be available on the Events page of the MongoDB investor relations website at https://investors.mongodb.com/events-and-presentations/events. A replay of the webcasts will also be available for a limited time.

About MongoDB
MongoDB is the leading modern, general purpose database platform, designed to unleash the power of software and data for developers and the applications they build. Headquartered in New York, MongoDB has more than 14,200 customers in over 100 countries. The MongoDB database platform has been downloaded over 70 million times and there have been more than one million MongoDB University registrations.

Investor Relations
Brian Denyeau
ICR for MongoDB
646-277-1251
ir@mongodb.com

Media Relations
Mark Wheeler
MongoDB
866-237-8815 x7186
communications@mongodb.com

View original content to download multimedia:http://www.prnewswire.com/news-releases/mongodb-inc-to-present-at-the-citi-2019-global-technology-conference-and-the-deutsche-bank-2019-technology-conference-300909780.html

SOURCE MongoDB

Article originally posted on InfoQ. Visit InfoQ

At KubeCon + CloudNativeCon China 2019, held in Shanghai and attended by over 3500 attendees, the major talk was about the pace of growth of the Kubernetes community and discussions were focused on how to contribute to the community.

In addition to talks by the local Chinese community, many of the event’s presentations touched on the staggering numbers that the Kubernetes community has managed to collect in a relatively short amount of time. Lots of the hallway conversations with developers were focused on how contributions can be made in some form to the community, as the feeling of merely being part of the community was not enough. The Cloud Native Computing Foundation published some stats recently that track the Kubernetes journey which is the foundation’s largest hosted project.

InfoQ caught up with Nikhita Raghunath, a contributor to the Kubernetes community. She is also a CNCF Ambassador and the technical lead for SIG Contributor Experience.

Raghunath begins the discussion by outlining her first contribution, and although it initially appeared to be an intimidating experience her fears were eased by her interactions with the community. She also talks about the organization of the community, and outlined some of the tools in the contributors aresenal with a key goal of enabling a better understanding to the reader of how and where to contribute.

Although people can read the detailed Contributor Guide for a comprehensive how-to, Raghunath outlined several quick tips and tricks that are especially focused on the impatient contributors who are keen to get started and to move up the contributor ladder.

InfoQ: Welcome to InfoQ. Can you start by highlighting your first contribution, please. How did you plan this, how long did it take and any lessons learned?

Raghunath: When I was in college, I noticed that Kubernetes was participating in the Google Summer of Code (GSoC) internship program. GSoC is a four month paid internship program where students can work on open source projects.

As part of my GSoC application, I decided to work a bug related to incorrect marshaling of integers in CRDs (called TPRs back then). I wasn’t really familiar with how TPRs worked internally, so I decided to first read the design proposal related to it. Once I got a fair idea about it, I then moved onto understand how Kubernetes did JSON serialization and deserialization. To be honest, understanding this from scratch was a little intimidating. I looked at existing issues, grepped the codebase for terms related to it and asked numerous questions in the SIG API Machinery slack channel. After this, I finally created a PR and it got merged!

The whole process took me about a week, with a couple of hours of work each day. I remember being confused about how the whole GitHub automation worked and I was afraid that I’d be asking really silly questions in Slack. Looking back, I realize how my fears were unfounded because everyone was really kind and patient while answering. This was in 2017 and we have come a long way in terms of contributor-facing documentation since then. There are docs explaining the automation and videos about code-base tours. My biggest suggestion would be not shy away from asking questions and show up on slack and (if possible) meetings so that other contributors see that you are active and know who you are. This also helps while taking up/delegating new tasks and moving up the contribution ladder to a reviewer.

InfoQ: Can you offer any tips on how to get started with contributing, especially those that are impatient and don’t want to read the contributor guide from cover to cover?

Raghunath: We have a cheatsheet (in eight languages) for this very reason! 🙂

Apart from the cheatsheet, here are some practical tips:

• Start with lurking in SIG slack channels and meetings! It removes any added pressure about introducing yourself or contributing from the beginning. It would also prove beneficial in understanding the community conventions and the current tasks the SIG is working on. The contributor calendar contains information about all SIG meetings. In case you are not able to attend a meeting, meeting notes can be helpful in getting a quick rundown.
• Kubernetes uses Prow for CI/CD and GitHub automation in the form of “slash commands”. When you create a PR, you might see bots adding certain labels and comments and even asking you to run certain commands. Understanding this automation is crucial to make sure that your PR gets adequate attention from reviewers and is merged quickly. The command help page gives a handy overview of all commands including examples on how to use them.
• To find an issue to work on, you can look for issues having the `good-first-issue` or the `help-wanted` labels. Apart from these, fixing lint, shellcheck or static check errors are good places to start diving in with code contributions since they don’t require a lot of familiarity with the codebase.
• For an overview of the kubernetes/kubernetes repo, you can checkout the codebase tour. If you’d like to have a codebase tour of specific repos, you can also ask about them in the Meet Our Contributors sessions. Running a `git blame` on a particular file can point you to the specific commit/PR that led to the change. This is very helpful since PRs contain a lot of additional context and discussions.
• If you are looking for specific mentorship and are willing to contribute a certain number of hours per week, consider “shadowing” on the release team. Shadows get mentored by leads, and can even step up to become leads in future release cycles!

InfoQ: Is it significantly easier to contribute to the Kubernetes docs than to an actual Kubernetes code repository? Can you provide some comparisons of contributing to docs versus contributing to code repositories?

Raghunath: I believe this is more nuanced. Fixing typos in documentation are easier for sure, but doc contributions extend far beyond that. Expressing technical concepts in clear, concise and unambiguous terms such that it’s easily understandable by people with all levels of technical background is tricky to get right.

One of the tasks actively being worked on by SIG Docs right now is localization to different languages. While this might seem easier on the surface, it is much more strenuous. Localization goes one step beyond just translation and requires that all concepts be adapted to convey a similar connotation in the target language, while still maintaining the original meaning. Implementing this in practice requires careful review and a vast understanding of the technical concepts and the language.

Let’s compare this with code changes like fixing lint, shellcheck or static check errors. Fixing these involve running a script, which then outputs failures containing exact reasons for the errors. Since the reasons are already known (most of the times, the reasons also contain suggestions on the required changes), it is easier to fix them. Moreover, fixing these errors don’t particularly require an immense knowledge about technical Kubernetes concepts.

InfoQ: If I know how to issue a Pull Request (PRs) on GitHub, what else do I need to know from a technical perspective before I can start contributing code to code repositories?

Raghunath: Firstly, you will need to setup your development environment. Depending on the repository you are contributing to, you might need to run additional commands as well. The CONTRIBUTING.md file at the root of the repo would be a good place to find more information about these. Once you make changes to the code, it is important to split the changes into logical series of smaller commits. One quick tip is to have any generated code in a separate commit since this makes reviewing the changes easier.

While we don’t impose any specific restrictions on commit message style, adhering to good commit message practices is always helpful. Note that certain repos might also impose additional restrictions like disallowing merge commits or keywords in commit messages. After creating the PR, you will notice bots adding labels and comments to your PR. Understanding the GitHub automation and slash commands as mentioned above will help in interacting with bots easier. Here are some commands that are commonly used:

• Apply the SIG label by adding a comment /sig cli. This will make sure that your PR is surfaced to SIG CLI reviewers.
• Writing /cc @username will request reviews and /assign @username will assign a particular person to your PR.
• /retest will automatically retest any failed tests on your PRs. This is especially handy in case your PR encounters any flakes.
• And the most important command: /meow and /woof will make the bot comment with a cat and dog picture!

After your PR looks good to the reviewer, they will add the “lgtm” (looks good to me) and “approved” labels to your PR using the /lgtm and /approve commands. Once your PR has these labels, it will get automatically merged by the bot.

InfoQ: Are there any particular Kubernetes communities that are more amenable for making contributions? In contrast, which ones are more challenging to contribute to?

Raghunath: There are certain tricky bugs like scalability failures or flaky tests that are harder to debug and fix but in my opinion, it is not possible to label a particular repo as harder to contribute to with a blanket statement.

Having said this, many people equate contributing to Kubernetes as contributing to the core kubernetes/kubernetes repo. But this is far from the truth. There are many other subprojects and repos in the `kubernetes-sigs`, `kuberentes-client` and the `kubernetes-csi` GitHub orgs as well. All of these repos are incredibly important and generally have a smaller set of folks working on them. Working on these subprojects is “easier” in the sense that you can have greater interactions with the maintainers and have quicker review response times.

InfoQ: Besides the satisfaction of contributing code to open source, are there other rewards to contributing code in the Kubernetes community?

Raghunath: Code was the initial aspect that attracted me, but I stayed even after my internship and continued contributing because of the community. “Come for the code, stay for the people” definitely resonates with me. I made countless friends from various parts of the world, and while many of them work at different companies, we still get to work together and learn from each other.

Personally, the biggest reward for me was that a lot of doors opened for me while I was hunting for jobs. Several contributors have also switched jobs, and yet continue to work on Kubernetes same as before. This helps in individual growth and also makes the project succeed long-term.

And of course, there are other rewards like the Kubernetes contributor patch, a chance to attend the Kubernetes Contributor Summit (co-located with KubeCons) and a big stash of Kuberentes-related swag!

In summary, although there a number of steps and processes associated with contributing to the Kubernetes community, understanding and following the guidance outlined above may reduce some of the intimidation factor, and make contributing to the community a lot easier.

More detailed information on contributing to the Kubernetes community is available in the Contributor guide.

Article originally posted on InfoQ. Visit InfoQ

Chirp uses audio to send and receive data using only a device’s speaker and microphone. Recently, Chirp had a chance to test their technology by sending signals to the Moon. InfoQ has spoken with Daniel Jones, Chief Technology Officer at Chirp, to learn more about Chirp codes.

Chirp sends data using either audible or inaudible frequencies, enabling the exchange of data between devices without requiring the establishment of a network or pairing. The devices exchanging Chirp codes can share the same acoustic environment, e.g., a room, or use any medium able to carry audio, such as a telephone line. Given the intrinsic nature of audio, Chirp codes can be easily broadcast to multiple devices at hearing distance.

On the technical side, Chirp relies on frequency shift keying (FSK) for signal modulation. Chirp messages can include any specific sequences that are used to indicate the start and length of a message as well as support error detection and correction.

As mentioned, Chirp recently carried through an experiment and sent a Chirp audio code to The Moon:

The signal was heard from the computer speaker, followed by a moment’s delay as it was beamed from the radio telescope and upwards through the earth’s atmosphere. The moon-bounced response then crackled back through the Skype link, backed with cosmic static. Astonishingly, given the number of lossy links in the transmission pipeline, the transmission decoded first time.

The major advantages of data-over-audio technologies like Chirp are the ability to let devices communicate without the hassle of establishing a network setup or pairing. In addition, audio codes are useful in environments forbidding the use of radio-frequencies.

InfoQ has spoken with Daniel Jones, Chief Technology Officer at Chirp, to learn more about Chirp codes.

Could you explain a few compelling use cases where Chirp audio codes could be used? What kind of existing technology do they improve upon or aim to replace?

Although there’s an array of connectivity technologies out there, data-over-sound fits a niche that no other technology quite fulfils: short-range, multi-way data transfer that happens at the tap of a button. Chirp’s USP is that it just works: no passwords or pairing, no need to enable Bluetooth, no problems with device support.

Another benefit of using sound as a medium is that it won’t carry through walls, unlike RF-based media like Bluetooth and Wi-Fi. That makes it suitable for tasks like room presence detection, in common scenarios like shared workspaces where you might have meeting room hubs in lots of adjoining rooms.

We’re working with providers to make meeting room hubs that send out Chirp ultrasonic beacons that can only be heard within that room, which contain all of the network information. You step into the room, your device detects exactly what room you’re in, receives the Wi-Fi credentials, connects to the network, and then connects to the meeting room screen. This is one of our favourite application areas as it solves so many pain points at once.

We’re also seeing a lot of uptake in IoT. One common use case is configuring smart devices with Wi-Fi credentials; instead of having to manually enter your password into each new smart device that you bring into your home, you can simply chirp them from your mobile device.

Chirp is also popular in toys and gaming, for tasks such as discovering nearby gamers and creating data links between the TV and accompanying second-screen devices.

What are the advantages of using inaudible audio codes for device-to-device communication?

The key advantages are:

• It’s friction-less: Chirp doesn’t require any passwords or pairing. Any device that hears the tone receives the data.

• It’s low-power and low-cost. Chirp is provided as a software-only solution, so if your device already has audio I/O, you won’t need to make any changes to your bill of material.

• It’s universal: It runs on any mobile device made in the past 10 years, and can even be transmitted down simple channels like VoIP or streaming video.

• It won’t pass through walls, which enables us to do room presence detection outlined above.

• It’s secure and trustworthy: As it doesn’t pass any data to the cloud, it is safe from remote snoopers. You can layer industry-standard cryptography algorithms for private transactions.

What are the current limits of Chirp technology in terms of throughput, energy efficiency, reliability, overall quantity of data that can be exchanged?

Reliability has always been our key objective, as it’s so important for that friction-less experience. We’ve dedicated years of research time to solving these problems. And thanks to the optimisation work of our expert embedded engineers, Chirp requires very little power. It can run on sub-$1 microchips (Arm Cortex-M4 @ 64Mhz), and, paired with wake-on-sound microphones such as Vesper’s VM1010, uses less power than BLE.

Since the acoustic channel is very noisy, we must highlight that you’re not likely to see data rates anywhere near those of BLE. At typical peer-to-peer ranges, expect rates of 100-200bps. That’s enough to send an email address, token ID, or an encrypted set of credentials – but maybe not your home videos! At NFC range, we’re able to transmit at 1kbps.

What were/are the major technological stumble points you needed to solve to make Chirp codes work reliably?

For the first few years, we were firmly focused on the R&D necessary to make data-over-sound robust enough for noisy real-world environments. We began in a research lab in University College London, doing field tests in all sorts of noisy places – pubs, streets, rock concerts.

This was tested to its limits when we began trialling with EDF Energy on creating long-range ultrasonic links within their nuclear power stations, to solve the challenges they have with Wi-Fi restrictions in these sensitive environments. Remarkably, we were able to attain 100% data integrity over a long-duration, long-range test, even in the deafening 100dB surrounds of their turbine hall.

More recently, we’ve been porting the code to work on a wide range of IoT devices and focusing on power and optimisation. This has opened up the technology to a lot of new devices and form factors, most recently the new Arduino Nano 33 BLE Sense board.

Could you explain how the “Moon mission” came about? In what ways is your tech relevant to space missions? And what is the meaning of being able to send an audio code to The Moon and then decode it?

We were one of the technical sponsors of the first hackathon at Abbey Road Studios, where we met a talented artist called Martina Zelenika. She has an ongoing art project in collaboration with a radio observatory in the Netherlands, in which she bounces short fragments of audio off the surface of The Moon, encoded as RF signals, and decodes the reflected signal back here on earth.

We thought it would be wonderful to send the first Chirp signal to The Moon and back. And, in a testament to the decoder’s reliability, we were able to decode the noisy reflected signal first time.

It would be of limited use in real space missions as mature transmission schemes are already used which modulate data directly onto RF waves. In our version, we were sending data over sound over radio, which — although a fun experiment — is unlikely to be adopted by NASA any day soon!

Chirp is not the only data-over-sound solutions provider. Other players in the market include ITC Infotech, LISNR, Sonarax, and others.

Chirp provides a suite of cross-platform enterprise-ready SDKs that can be downloaded for free. Written in C, Chirp SDK supports Arduino, Android, iOS, Windows, all major browsers, and desktop OSes.

Article originally posted on InfoQ. Visit InfoQ

Cloud provider DigitalOcean recently released a pair of new managed data services. Their Managed MySQL and Redis offerings are on-demand and elastic, and offer a variety of sizes and high-availablity options.

In February of 2019, DigitalOcean launched their managed database service with PostgreSQL as the first supported engine. They shared that MySQL and Redis options were on the roadmap, and this month they delivered on that commitment. In a blog post about their Managed Databases for MySQL and Redis, André Bearfield of DigitalOcean explained how these databases reflect the simplicity and usability that’s synonymous with their services.

 Developers of all skill levels, even those with no prior experience in databases, can spin up database clusters with just a few clicks. Select the database engine, storage, vCPU, memory, and standby nodes and we take care of the rest. 

DigitalOcean delivered capabilities one might expect from a managed database. They offer automatic software updates to the database engine and underlying operating system, in a maintenance window of your choosing. There are automatic daily backups that are retained for seven days. During instance failure, DigitalOcean automatically replaces failed nodes with new ones that pick up from the point of failure. For manual recovery from a backup, users restore into a new database instance. Both Managed MySQL and Redis options support up to two standby nodes that take over automatically if the primary node fails. Managed MySQL customers can provision read-only nodes in additional geographic regions for horizontal scaling. Managed MySQL customers also get access to monitoring and proactive alerting functionality, and the ability to fork an entire cluster based on a specific point in time. Bearfield says that Managed Redis will also get database metrics and monitoring upon general availability.

Both the Managed MySQL and Redis offerings come with two cluster types: single node or high-availability. The single node clusters start at $15 per month and provide 1 GB of memory, 1 vCPU, and 10 GB of SSD disk storage. As evident by the name, the single node clusters aren’t highly available, but do support automatic failover. The high availability clusters offer up to two standby nodes and begin at $50 per month. The single node plan offers database instances as large as 32 GB of RAM, 8 vCPUs, and 580 GB of storage. The high availability plan goes a notch larger with machines offering 64 GB of RAM, 16 vCPUs, and 1.12 TB of disk storage.

Whether you look at the 2019 Stack Overflow developer survey, Jetbrains 2019 Dev Ecosystem survey, or DBEngine rankings, it’s clear that MySQL and Redis are wildly popular database engines. Shiven Ramji, DigitalOcean’s VP of Product, says that the company is driven by supplying developers with what they need.

“With the additions of MySQL and Redis, DigitalOcean now supports three of the most requested database offerings, making it easier for developers to build and run applications, rather than spending time on complex management,” said Shiven Ramji, DigitalOcean’s senior VP of Product. “The developer is not just the DNA of DigitalOcean, but the reason for much of the company’s success. We must continue to build on this success and support developers with the services they need most on their journey towards simple app development.”

The Managed Databases for MySQL and Redis are currently available in a subset of regions globally, with DigitalOcean promising wider availability soon. For both engines, you’re allowed up to 3 clusters per account, private networking is only available within the same datacenter region, and while clusters can be resized, they can only be made larger, not smaller. MySQL v8 is the supported edition with a handful of unsupported features, and Redis version 5 is offered with a few limitations. 

During the past few years, DigitalOcean has expanded beyond offering low-cost virtual private servers. They’ve launched an object storage service, managed Kubernetes offering, more robust networking options, and now managed databases.

Article originally posted on InfoQ. Visit InfoQ

In a recent Harvard Business Review article, “Why inclusive leaders are good for organizations and how to become one”, authors Juliet Bourke and Andrea Espedido describe their new research into the characteristics and benefits of ‘inclusive’ leadership. Findings, including Improved collaboration and improved activation of team diversity, are extremely useful for software teams striving for Agile and DevOps cultures.

Inclusive leadership focuses on multidisciplinary teams that combine peoples’ collective capabilities. Bourke and Espedido found that:

Teams with inclusive leaders are 17% more likely to report that they are high performing, 20% more likely to say they make high-quality decisions, and 29% more likely to report behaving collaboratively. What’s more, we found that a 10% improvement in perceptions of inclusion increases work attendance by almost 1 day a year per employee, reducing the cost of absenteeism.

The authors discovered six traits or behaviors that distinguished inclusive leaders from others:

Visible commitment: They articulate authentic commitment to diversity, challenge the status quo, hold others accountable and make diversity and inclusion a personal priority.

Humility: They are modest about capabilities, admit mistakes, and create the space for others to contribute.

Awareness of bias: They show awareness of personal blind spots as well as flaws in the system and work hard to ensure meritocracy.

Curiosity about others: They demonstrate an open mindset and deep curiosity about others, listen without judgment, and seek with empathy to understand those around them.

Cultural intelligence: They are attentive to others’ cultures and adapt as required.

Effective collaboration: They empower others, pay attention to diversity of thinking and psychological safety, and focus on team cohesion.

The article also includes links to the main research published previously by the authors and some quick advice on what to do. Their four ways to get started are:

• Know your inclusive-leadership shadow: Seek feedback
• Be visible and vocal
• Deliberately seek out difference
• Check your impact, are people copying your role modeling?

The sample for the study is from three Australian organisations, This single aspect of leadership should be of great help to those organisations striving for more impact from their investment in diversity. 

The article cites additional sources for further investigation that Deloitte have published:

Article originally posted on Data Science Central. Visit Data Science Central

When Anthem first announced the breach of 80 million of its records, FBI spokespersons credited Anthem with responding only weeks after the attack started.

Soon after, Brian Krebs, who blogs at KrebsOnSecurity, uncovered evidence that the attack might have started much earlier.

Why does it take so long for companies like Anthem to discover data breaches in progress and even longer to tell consumers?

Although the answer to that question remains complicated, one truth is undeniably simple: Consumers are the last to know.

10 Months in the Making

On April 21, 2014, the IP address 198.200.45.112 became home to a new domain called we11point.com. Additional subdomains, including myhr.we11point.com, hrsolutions.we11point.com, and excitrix.we11point.com were added. Security discovered that the subdomains looked like the components of the Wellpoint corporate network. Wellpoint, until late 2014, was Anthem’s corporate name.

The last subdomain, excitrix.we11point.com, is associated with a piece of backdoor malware that masquerades as Citrix’s VPN software. The malware had a security certificate signed by DTOPTOOLZ.COM, which is a known signature for a large Chinese cyber-espionage group called Deep Panda. In May 2014, Deep Panda targeted a series of phishing emails toward Wellpoint employees, trying to obtain their login credentials.

Interestingly, Premera Blue Cross, another large health insurance provider, disclosed a data breach on March 15, 2015. A few weeks before the disclosure, a security firm tied Deep Panda to another domain called prennera.com. Premera discovered its breach in January 2015 but revealed that the first attack happened back in May 2014.

Experts say that Deep Panda probably didn’t want to steal and sell 80 million records worth of patient information. Instead, the organization wanted to find records on one or more individuals that could help it infiltrate a U.S. defense program.

Why it Went Unnoticed

Attacks go unnoticed for several different reasons. Because few attacks disrupt a company’s external services, intruders often remain undetected on the network for long periods of time. Also, companies might receive endpoint alerts related to intrusions, but they receive so much security data that they can’t analyze all of it.

Buried by alert data and logs, analysts can’t distinguish which notifications are real and which are false alarms. Time and financial constraints prevent them from investigating every lead and keeps big companies like Anthem from connecting the dots.

Despite the staggering evidence that companies are typically slow to detect data breaches, corporate security teams think they’re better than they are. For instance, a security survey of 500 IT decision makers revealed that when asked, IT leaders claimed that they discovered most data breaches within 10 hours. The Verizon Data Breach Investigation Report, however, contradicts their claims.

According to Verizon, 66 percent of breaches take months or even years to detect. Properly configured network and cloud security and data center defense tools can prevent many infiltrations. However, when attackers obtain an administrator-level password, as Deep Panda did during the Anthem attack, they don’t have to break through company defenses.

Disclosing the Breach to Consumers

Because Congress has so much difficulty coming to a consensus on cybersecurity, particularly data breach disclosure requirements, different states have created a mashup of laws. Instead of containing specific timeframes regarding data breach disclosure, these laws often use vague wording. For example, a New York disclosure law states that companies must publish “in the most expedient time possible and without unreasonable delay,” but it also allows delays that accommodate “the legitimate needs of law enforcement.”

Anthem, possibly learning from the mistakes of other organizations, disclosed its breach and contacted law enforcement not long after the initial discovery. Other companies aren’t so forthcoming. They keep breaches under wraps for months before letting customers know their information was compromised.

Making Prevention a Priority

Instead of focusing on fending off attacks as they happen, companies need threat intelligence tools that identify internal vulnerabilities and data assets. Then, they should compare that with data from the external threat landscape to anticipate the company’s likelihood of being attacked.

Congress introduced the Data Security and Breach Notification Act of 2015, but critics argue that the law would weaken stronger state laws that are favorable to consumers. Before it can pass a law, Congress has to balance the consumer’s right to know with the company’s desire to keep information private. Staying quiet might help an ongoing law enforcement investigation, but it shouldn’t be used to save face.

 

Article originally posted on InfoQ. Visit InfoQ

Earlier this month Microsoft released .NET Core 3.0 Preview 8 for Windows, macOS, and Linux.

The new release contains bug fixes and enhancements for the ASP.NET Core, CoreFX, and CoreCLR product areas. There are no new features since the framework entered the freezing period in the previous release, and all development efforts are focused on stability and reliability.

ASP.NET Core was the product area most affected by this release. Some of the project templates were updated, like the Angular template (updated to Angular 8). Blazor logging and diagnostics capabilities were improved, along with gRPC diagnostics. Other enhancements include the addition of new networking primitives for non-HTTP servers, case-sensitive component binding, and culture-aware data binding. More details on the ASP.NET Core updates can be found here.

CoreFX received multiple bug fixes and enhancements for the System.Text.Json API, while Core CLR received bug fixes related to diagnostics and exception handling in Linux environments. A complete list of all bug fixes and releases for all affected product areas can be found here.

This preview of .NET Core 3.0 is supported by Microsoft and can be used in production: the Microsoft .NET Site has already been updated with the new release. With the final 3.0 release scheduled for next month, Microsoft officially recommends the start of adoption planning for .NET Core 3.0.

Article originally posted on Data Science Central. Visit Data Science Central

---

---

---

---

---

---

---

---

---

---

---

---

Article originally posted on Data Science Central. Visit Data Science Central

The Ugly Truth Behind All That Data

We are in the age of data. In recent years, many companies have already started collecting large amounts of data about their business. On the other hand, many companies are just starting now. If you are working in one of these companies, you might be wondering what can be done with all that data.

 

What about using the data to train a supervised machine learning (ML) algorithm? The ML algorithm could perform the same classification task a human would, just so much faster! It could reduce cost and inefficiencies. It could work on your blended data, like images, text documents, and just simple numbers. It could do all those things and even get you that edge over the competition.

 

However, before you can train any decent supervised model, you need ground truth data. Usually, supervised ML models are trained on old data records that are already somehow labeled. The trained models are then applied to run label predictions on new data. And this is the ugly truth: Before proceeding with any model training, any classification problem definition, or any further enthusiasm in gathering data, you need a sufficiently large set of correctly labeled data records to describe your problem. And data labeling — especially in a sufficiently large amount — is … expensive.

 

By now, you will have quickly done the math and realized how much money or time (or both) it would actually take to manually label all the data. Some data are relatively easy to label and require little domain knowledge and expertise. But they still require lots of time from less qualified labelers. Other data require very precise (and expensive) expertise of that industry domain, likely involving months of work, expensive software, and finally, some complex bureaucracy to make the data accessible to the domain experts. The problem moves from merely expensive to prohibitively expensive. As do your dreams of using your company data to train a supervised machine learning model.

 

Unless you did some research and came across a concept called “active learning,” a special instance of machine learning that might be of help to solve your label scarcity problem.

 

What Is Active Learning?

Active learning is a procedure to manually label just a subset of the available data and infer the remaining labels automatically using a machine learning model.

 

The selected machine learning model is trained on the available, manually labeled data and then applied to the remaining data to automatically define their labels. The quality of the model is evaluated on a test set that has been extracted from the available labeled data. If the model quality is deemed sufficiently accurate, the inferred class labels extended to the unlabeled data are accepted. Otherwise, an additional subset of new data is extracted, manually labeled, and the model retrained. Since the initial subset of labeled data might not be enough to fully train a machine learning model, a few iterations of this manual labeling step might be required. At each iteration, a new subset of data to be manually labeled needs to be identified.

 

As in human-in-the-loop analytics, active learning is about adding the human to label data manually between different iterations of the model training process (Fig. 1). Here, human and model each take turns in classifying, i.e., labeling, unlabeled instances of the data, repeating the following steps.

 

Step a –Manual labeling of a subset of data

At the beginning of each iteration, a new subset of data is labeled manually. The user needs to inspect the data and understand them. This can be facilitated by proper data visualization.

Step b –Model training and evaluation

Next, the model is retrained on the entire set of available labels. The trained model is then applied to predict the labels of all remaining unlabeled data points. The accuracy of the model is computed via averaging over a cross-validation loop on the same training set. In the beginning, the accuracy value might oscillate considerably as the model is still learning based on only a few data points. When the accuracy stabilizes around a value higher than the frequency of the most frequent class and the accuracy value no longer increases — no matter how many more data records are labeled — then this active learning procedure can stop.

 

Step c –Data sampling

Let’s see now how, at each iteration, another subset of data is extracted for manual labeling. There are different ways to perform this step (query-by-committee, expected model change, expected error reduction, etc.), however, the simplest and most popular strategy is uncertainty sampling.

 

This technique is based on the following concept: Human input is fundamental when the model is uncertain. This situation of uncertainty occurs when the model is facing an unseen scenario where none of the known patterns match. This is where labeling help from a human — the user — can change the game. Not only does this provide additional labels, but it provides labels for data the model has never seen. When performing uncertainty sampling, the model might need help at the start of the procedure to classify even simple cases, as the model is still learning the basics and has a lot of uncertainty. However, after some iterations, the model will need human input only for statistically more rare and complex cases.

 

After this step c, we always start again from the beginning, step a. This sequence of steps will take place until the user decides to stop. This usually happens when the model cannot be improved by adding more labels.

 

Why do we need such a complex procedure as active learning?

Well, the short answer is: to save time and money. The alternative would probably be to hire more people and label the entire dataset manually. In comparison, labeling instances using an active learning approach is, of course, more efficient.

 

 

Uncertainty Sampling

Let’s have a closer look now at the uncertainty sampling procedure.

 

As for a good student, it is more useful to clarify what is unclear rather than repeating what the student has already assimilated. Similarly, it is more useful to add manual labels to the data that the model cannot classify confidently than to the data where the model is already confident.

 

Data where the model outputs different labels with comparable probabilities are the data about which the model is uncertain. For example, in a binary classification problem, the most uncertain instances are those with a classification probability of around 50% for both classes. In a multi-classification problem, highest uncertainty predictions happen when all class probabilities are close. This can be measured via the entropy formula from information theory or, better yet, a normalized version of the entropy score.

 

 

  

Let’s consider two different data rows feeding a 3-class classification model. The first row was predicted to belong to class 1 (label 1) with 90% probability and to class 2 and class 3 with only 5% probability. The prediction here is clear: label 1. The second data row, however, has been assigned a probability of belonging to all three labels of 33%. Here the class attribution is more complicated.

 

Let’s measure their entropy. Data in Row1 has a higher entropy value than data in Row0 (Table 1), and this is not surprising. This selection via entropy score can work with any number n of classes. The only requirement is that the sum of the model probabilities always adds up to 1.

 

Summarizing, a good active learning system should extract all those rows for manual labeling that will benefit most from human expertise rather than more obvious scenarios. After a few iterations, the human-in-the-loop should find the selection of data rows for labeling less random and more unique.

 

Active Learning as a Guided Labeling Web Application

In this section, we would like to describe a preconfigured and free blueprint web application that implements the active learning procedure on text documents, using KNIME software and involving human labeling between one iteration and the next. Since it takes advantage of the Guided Analytics feature available with KNIME Software, it was named “Guided Labeling.”

 

The application offers a default dataset of movie reviews from Kaggle. For this article, we focus on a sentiment analysis task on this default dataset. The set of labels is therefore quite simple and includes only two: “good” and “bad.”

 

The Guided Labeling application consists of three stages (Fig. 2).

1. Data upload and label set definition. The user, our “human-in-the-loop,” starts the application and uploads the whole dataset of documents to be labeled and the set of labels to be applied (the ontology).
2. Active learning. This stage implements the active learning loop.

• Iteration after iteration, the user manually labels a subset of uncertain data rows.
• The selected machine learning model is subsequently trained and evaluated on the remaining subset of labeled data. The increase in model accuracy is monitored until it stabilizes and/or stops increasing.
• If the model quality is deemed not yet sufficient, a new subset of data containing the most uncertain predictions is extracted for the next round of manual labeling via uncertainty sampling.

3. Download of labeled dataset. Once it is decided that the model quality is sufficient, the whole labeled dataset — with labels by both human and model — is exported. The model is retrained one last time on all available instances, used to score documents that are still unlabeled, and is then made available for download for future deployments.

 

 

From an end user’s perspective, these three stages translate to the following sequence of web pages (Fig. 3).

 

 

In the first page, the end user has the possibility to upload the dataset and define the label set. The second page is an easy user interface for the quick manual labeling of the data subset from uncertainty sampling.

 

Notice that this second page can display a tag cloud of terms representative of the different classes. Tag clouds are a visualization used to quickly show the relevant terms in a long text that would be too cumbersome to read in full. We can use the terms in the tag cloud to quickly index documents that are likely to be labeled with the same class. Words are extracted from manually labeled documents belonging to the same class. The top most frequent 50 terms across classes are selected. Of those 50 terms, only the terms present in the still unlabeled documents are displayed in an interactive tag cloud and color coded depending on the document class.

 

There are two labeling options here:

• Label the uncertain documents one by one as they are presented in decreasing order of uncertainty. This is the classic way to proceed with labeling in an active learning cycle.
• Select one of the words in the tag cloud and proceed with labeling the related documents. This second option, while less traditional, allows the end user to save time. Let’s take the example of a sentiment analysis task: By selecting one “positive” word in the tag cloud, mostly “positive” reviews will surface in the list, and therefore, the labeling is quicker.

 

Note. This Guided Labeling application works only with text documents. However, this same three-stage approach can be applied to other data types too, for example, images or numerical tabular data.

 

Guided Labeling in Detail

Let’s check these three stages one by one from the end user point of view.

 

Stage 1: Data Upload and Label Set Definition

 

The first stage is the simplest of the three, but this does not make it less important. It consists of two parts (Fig. 4):

• uploading a CSV file containing text documents with only two features: “Title” and “Text.”
• defining the set of classes, e.g., “sick” and “healthy” for text diagnosis of medical records or “fake” and “real” for news articles. If too many possible classes exist we can upload a CSV file listing all the possible string values the label can assume.

 

Stage 2: Active Learning

It is now time to start the iterative manual labeling process.

 

To perform active learning, we need to complete three steps, just like in the diagram at the beginning of the article (Fig. 1).

 

Step 2a –Manual labeling of a subset of data

The subset of data to be labeled is extracted randomly and presented on the left side (Fig. 5.1 A) as a list of documents.

 

If this is the first iteration, no tag clouds are displayed, since no classes have been attributed. Let’s go ahead and, one after the other, select, read and label all documents as “good” or “bad” according to their sentiment (Fig. 5.1 B).

 

The legend displayed in the center shows the colors and labels to use. Only labeled documents will be saved and passed to the next model training phase. So, if a typo is detected or a document was skipped, this will not be included in the training set and will not affect our model. Once we are done with the manual labeling, we can click “Next” at the bottom to start the next step and move to the next iteration.

 

If this is not the first iteration anymore and if the selected machine learning model has already been trained, a tag cloud is created from the already labeled documents. The tag cloud can be used as a shortcut to search for meaningful documents to be labeled. By selecting a word, all those documents containing that word are listed. For example, the user can select the word “awful” in the word cloud and then go through all the related documents. They are all likely to be in need of a “bad” label (Fig 5.2)!

 

 

 

Step 2b –Training and evaluating an XGBoost model

Based on a subset of the few labeled documents to be used as a training set, an XGBoost model is trained to predict the sentiment of the documents. The model is also evaluated on the same labeled data. Finally, the model is applied to all data to produce a label prediction for each review document.

 

After labeling several documents, the user can see the accuracy of the model improving in a bar chart. When accuracy reaches the desired performance, the user can check the check box “Stop Labeling” at the top of the page, then hit the “Next” button and get to the application’s landing page.

 

Step 2c –Data sampling

Based on the model predictions, the entropy scorer is calculated for all yet unlabeled data rows; uncertainty sampling is applied to extract the best subset for the next phase of manual labeling. The whole procedure then restarts from step 2a.

 

Stage 3: Download of Labeled Dataset

We reached the end of the application. The end user can now download the labeled dataset, with both human and machine labels, and the model trained to label the dataset.

 

Two word clouds are made available for comparison: on the right, the word cloud of those documents labeled by the human in the loop and on the left, the word cloud of machine labeled documents. In both clouds, words are color coded by document label: red for “good” and purple for “bad.” If the model is performing a decent job at labeling new instances, the two word clouds should be similar and most words in them should have the same color (Fig. 6).

 

 

Guided Labeling for Active Learning

In this article, we wanted to illustrate how active learning can be used to label a full dataset while investing only a fractional amount of time in manual labeling. The idea of active learning is that we train a machine learning model well enough to be able to delegate it to the boring and expensive task of data labeling.

 

We have shown the three stages involved in an active learning procedure: manual labeling, model training and evaluation, and sampling more data to be labeled. We have also shown how to implement the corresponding user interface on a web-based application, including a few tricks to speed up the manual labeling effort using uncertainty sampling.

 

The example used in this article referred to a sentiment analysis task with just two classes (“good” and “bad”) on movie review documents. However, it could easily be extended to other tasks by changing the number and type of classes. For example, it could be used for topic detection for text documents if we provided a topic-based ontology of possible labels (Fig. 7). It could also be extended just as simply to other data types and classification tasks.

 

The Guided Labeling application was developed via a KNIME workflow (Fig. 8) with the free and open source tool KNIME Analytics Platform, and it can be downloaded for free from the KNIME Hub. If you need to perform active learning and label tons of data rows, we suggest downloading the blueprint workflow and customizing it to your needs. You could, for example, make it work for images, use another machine learning model, or implement some other strategy to train the underlying model.

 

It is now your turn to try out the Guided Labeling application yourself. See how easily and quickly data labeling can be done!

 

 

Paolo Tamagnini contributed to this article. He is a data scientist at KNIME, holds a master’s degree in data science from Sapienza University of Rome and has research experience from NYU in data visualization techniques for machine learning interpretability. Follow Paolo on LinkedIn.

For more information on KNIME, please visit www.knime.com and the KNIME blog.

Article originally posted on Data Science Central. Visit Data Science Central

Continued fractions are usually considered as a beautiful, curious mathematical topic, but with applications mostly theoretical and limited to math and number theory. Here we show how it can be used in applied business and economics contexts, leveraging the mathematical theory developed for continued fraction, to model and explain natural phenomena. 

The interest in this project started when analyzing sequences such as x(n) = { nq } = nq – INT(nq) where n = 1, 2, and so on, and q is an irrational number in [0, 1] called the seed. The brackets denote the fractional part function. The values x(n) are also in [0, 1] and get arbitrarily close to 0 and 1 infinitely often, and indeed arbitrarily close to any number in [0, 1] infinitely often. I became interested to see when it gets very close to 1, and more precisely, about the distribution of the arrival times t(n) of successive records. I was curious to compare these arrival times with those from truly random numbers, or from real-life time series such as temperature, stock market or gaming/sports data. Such arrival times are known to have an infinite expectation under stable conditions, though their medians always exist: after all, any record could be the final one, never to be surpassed again in the future. This always happens at some point with the sequence x(n), if q is a rational number — thus our focus on irrational seeds: they yield successive records that keep growing over and over, without end, although the gaps between successive records eventually grow very large, in a chaotic, unpredictable way, just like records in traditional time series.

See also my article  Distribution of Arrival Times for Extreme Events, here.  It turns out that the behavior of the records in the sequence x(n) can be different from those of random numbers, offering a new, broader class of statistical models.

1. Theoretical background (simplified)

In some sense, continued fractions provide the best rational approximations to irrational numbers such as q. The successive approximations, called convergents, converge to q. Let’s look at the sequence of successive records for q = SQRT(2)/2, to see how they are related to the denominators of its convergents in its continued fraction expansion. This is illustrated in the table below. 

If you do a Google search for  (say) “19601 114243”, the first search result is this: A001541 – OEIS. Interestingly, this sequence corresponds to denominators of convergents for SQRT(2)/2. In this case, the arrival times t(n) of the records have an explicit formula:

More details are available here. In general, if q is the square root of an integer (not a perfect square) the convergents can be computed explicitly: see here (PDF document). The interesting fact is that t(n), the arrival time of the n-th record, as n grows, is very well approximated by A * B^n  (here B^n means B at power n) where A and B are two constants depending on q. 

Now let turn to q = e = exp(1). The behavior is more chaotic, and pictured in the table below.  

Follow the red path! The first 20 convergents of e are listed below (source: here). Look at the denominator of each convergent to identify the pattern:

• 8 / 3,
• 11 / 4,
• 19 / 7,
• 87 / 32,
• 106 / 39,
• 193 / 71,
• 1264 / 465,
• 1457 / 536,
• 2721 / 1001,
• 23225 / 8544,
• 25946 / 9545,
• 49171 / 18089,
• 517656 / 190435,
• 566827 / 208524,
• 1084483 / 398959,
• 13580623 / 4996032,
• 14665106 / 5394991,
• 28245729 / 10391023,
• 410105312 / 150869313,
• 438351041 / 161260336

The same patterns applies to all irrational numbers. Indeed, we found an alternate way to compute the convergents, based on the records. 

2. Generalization and potential applications to real life problems

We found a connection between records (maxima) of some sequences, and continued fractions. Doing the same thing with minima will complete this analysis. Extreme events in business or economics settings are typically modeled using statistical distributions (Gumbel and so on), which did not live to their expectations in many instances.

The theory outlined here provides an alternative, with each number q serving as a particular model, with its own features distinct from all other numbers. In particular, the numerators and denominators of convergents of q can be used to model the arrival times of extreme events. While the records here are getting closer either to zero or to one, using a logarithm transformation allows you to model phenomena that are not bounded.

Note that the sequence x(n) has strong, long-range auto-correlations that are well studied (see appendix B in this book). One could try with the sequence x(n) = { b^n q } instead (b > 1) as it exhibits exponentially fast decaying auto-correlations, also studied in the same book. Even cross-correlations between two sequences have been studied: for instance, if x(n) = { nq } and y(n) = { nq * 12 / 35 }, then the correlation between the two sequences is 1 / (12 * 35). Replace 12 and 35 by any integers that do not share a common divisor, and as long as q is irrational, this result easily generalizes. For a proof, see here. This allows you to model multivariate time series and their cross-dependencies.

Finally, by googling some of the t(n) attached to q = log(3) / log(2) and discussed in the next section, I found that the associated convergents have some applications in music. See here for details. 

3. Original applications in music and probabilistic number theory

We focus here on q = log(3) / log(2). The music application is discussed here, and in this section we focus only on the number theory application. The purpose is to prove that the binary digits of SQRT(2) are uniformly distributed, with the proportion of 0 and 1 converging to 50/50 as you look at increasingly longer sequences of successive digits. The use of convergents here helps only with a very specific, narrow part of the proof, but a critical one (assuming a complete proof is ever found – to this day it is still a mystery.)

Let’s look at the sequence z(n) = 2^x(n). with x(n) = { nq } as usual. Since x(n) is equidistributed modulo one, it is uniformly distributed on [0, 1].  Thus the distribution of z(n) is that of 2^(-X) where X is uniformly distributed on [0, 1]. and its median is thus SQRT(2)/2. If n is odd, the empirical median of (z(1), z(2), …, z(n)) is the middle value when these values are sorted, so it is also one of these values. The empirical median converges to the theoretical median SQRT(2)/2. Also, each z(n) is a rational number: the length of its period in base 2 is equal to 2 * 3^(n-1). In addition, the proportion of 0 and 1 in the binary expansion of z(n) and thus in the empirical median regardless of n, is always exactly 50/50. While this is true for all n, it does not mean that this fact remains true for the limit of any converging infinite sub-sequence of z(n) values. Take the sub-sequence z(t(n)) with t(n) the arrival time of the records of x(n). That sub-sequence converges to 1/2. This is a counter-example, and the link to the continued fractions discussed earlier. Sub-sequences having that defect can and sometimes do converge to a non-normal number as in our example, and they must be excluded by carefully removing an infinite number of x(n) without impacting the median. That’s where these records and associated continued fractions become handy. 

But you don’t need to take all the x(n) into consideration. You can focus on a sub-sequence x(h(n)) where h(n) is an increasing sequence of integers, and as long as SQRT(h(n+1)) – SQRT(h(n)) tends to zero as n tends to infinity, the equidistribution theorem remains applicable, and thus the median will still be SQRT(2)/2. Hopefully one can find a sequence h(n) that avoids the problems mentioned in the previous paragraph. 

The table for t(n) for q = (log 3) / (log 2) can be found here.  Even though no one has ever fully proved that SQRT(2) has its binary digits uniformly distributed, a much deeper result is “known” (but of course, also unproved) and published here for the first time:

The proportion q(n) of binary digits of SQRT(2) (or any other normal number for that matter, say Pi or log 2), among the first n digits, satisfies

See here for details. In short, this is a consequence of a refined version of the Berry-Esseen theorem, itself a second-order approximation to the central limit theorem. It is true if the digits in question were distributed uniformly and independently with a Bernouilli distribution of parameter 1/2, as they appear to be. Empirical evidence seems to confirm this conjecture: the chart below shows the first 1,000,000 values of SQRT(n) * |q(n) – 1/2|.

To not miss our future articles (more on this topic coming soon), subscribe to our newsletter, here.