Month: March 2023

Article originally posted on InfoQ. Visit InfoQ

Transcript

Pilgrim: My name is Peter Pilgrim. I’m going to be talking to you about this, software engineering, sustainable empathic capacities, an excursion on the lived experience to minor creative genius. I am a Java champion, a senior manager at Cognizant. Previously, I was an independent contractor. I had this expansive view of software engineering. The first school of thought is that I would give an autobiographic view of my lived experience in software engineering. The second school of thought is that we are journeying, at least in my humble opinion, to empathy, to express empathy within our discipline.

Remote Sacking/Firing

First a corporate statement. The company has made a decision that the future of its vehicles going forward will be crewed by a third-party crew provider. Therefore, I’m sorry to say that this need to inform and employ you is terminated with immediate effect upon granted redundancy. Your final day of employment is today. Remote sackings and firings have happened so recently. British people wouldn’t have heard of better.com, but Americans, and Canadians, and North Americans have definitely heard of Vishal Garg, who fired over 900 employees over Zoom, in the 7th of December. How heartless and lacking in empathy was this, in the month of December? Yes, that’s it. Then in 17th of March, 2022, in Britain, on these shores, P&O Ferries dismissed 800 members of its shipping staff primarily from the Port of Dover, and that the British government didn’t take too well with this and the ramifications are still happening now.

What Is Empathy?

What is this thing that we are talking about? What is empathy? Where does it come from? This is a picture of the Homo sapien brain. It turns out, it’s a concoction. It’s from a German word, Einfühlung, which is a German psychological term, literally meaning feeling-in, inner, inside, intrinsic, the inner feeling. Germans are well known, famous for putting and inventing words from two separate words, so combining Ein, which is inner, and fühlung, which is feeling, to have this feeling in, or what we translate now, thanks to two scientists at Cambridge and Cornell University in 1908, as Empathos. Em is for in. Pathos is for feeling. That has been the early translation. This is why we have the word empathy in the English language. Thanks to Susan Lanzoni of Cambridge University Press.

Now for the technical details, or rather, the medical details. What seems to happen is we have an organ deep inside our brain, called the amygdala. This thing that looks like a grape or a squash grape, and we actually have two of this, because this picture is a cross section of the brain. We have our famous gray matter here, or the lobes, the frontal and parietal lobe, and the occipital lobe here, and the famous cerebellum, where we do a lot of our processing at the back of our brain. What seems to happen is that we have five sensitives, and I will concentrate just on one, the visual sense, when a photon hits the retina, a signal is transferred straight through our brain from left to right. Reach in and passing through the hypothalamus, which is this orange-yellow bit that looks like a bug’s head, looks like a squash potato, reaches the back of the occipital lobe here. Enters the cerebellum, and then magically, these signals are transferred via the hippocampus to the amygdala.

What happened recently, it’s been known since the 1960s, is that these signals hit the amygdala very quickly, so quickly they’re known as microexpressions or microsignals. They are the result of, I suppose, voluntary and involuntary emotional response conflicting with another human being. If you think about it, this is the very essence of communication. Here’s another human being, is threatening me, is he friendly, is he foe? What is it? We have to develop these social behaviors. This is because in the purple section here, we have an ancient part of our brain when we first cross-evolved through the millennia, or millions of years from amphibians that first called out of sea. We had lizard brains. What has happened through evolutions, we’ve kept on extending this brain structure, which we need.

It seems the amygdala is the heart of fear and flight anxiety. It’s part of our lizard brain, the limbic system. It’s responsible for connections, emotional learning, and memory regulation. That’s where the cerebellum comes in. It’s very complicated, and scientists are still trying to work out what is going on there. Essentially, emotions and empathy, and any chance of empathy comes from this little organ that looked like a grape or two grapes. Because we have one in the left and one on the right side of our brainstem, in our brains. We have two of them, just like two kidneys.

This is Labi Siffre, the famous singer, songwriter. I was listening to him a couple of days ago in fact, on a radio program, an interview he gave. I just quoted him here. “Humans tend to hate nuance.” I wonder why, especially in 2022. “We want to put things in a box, we want to compartmentalize. However, the reality is that there’s nothing simple about anything we do with human beings.” I love this next part. “We are complexity upon complexity upon paradox on toast.” That describes us. We have toast, the amygdala, and it’s pure. Because we have these lobes and this interpretation, we mix it all up, together. This is from the brilliant Labi Siffre talking about his musical performance, and what he thinks about musical performance. That is from the BBC, Colin Murray show, Midnight Meets, from the 4th of May, 2022.

I love this quote from Vex King. “It is your subconscious mind that is responsible for your beliefs. All that you perceive is a result of what you accept as true in the conscious mind.” You can choose to bury certain facts about the world, or have a total different independent view of the world for most human beings and what is deemed as reality. I would get vexed by Vex King, but you must live in a real world. How many people have had that said to them? The real world is what you actually believe to be true. You never suffer bullies or people when they force or coerce you into accepting something else that you know is true. That is not the way to live.

For me, as I was talking in the QCon London conference there, I had this observation in my life that, a change of contradictions, a series of disruptions that invalidates the assumptions or surroundings is a rug pull. I learned from my physics teacher who said, any news of the world that breaks the illusion, a system in a steady state condition is a whirlpool. My old physics teacher is proven right time over time. This isn’t me doing a somersault or the backflip, but it’s the start of my lived in experience. I went to a secondary school called Spencer Park in South London. Just like Harry Potter, we wore Blazers to school, in a secondary school. I have this coat of arms, Dieu defend le droit, defend the right, yours with God, defends the right, on my blazer here.

Feeling for Another’s Suffering/Vulnerability

I think at 13 years old, I finally figured out what I wanted to be in life. The first thing that happened is that I discovered Research Machines 380Z in the 1980s, because I joined the electronics club, [inaudible 00:10:55] and I, and Pete went to the computer club, and learned BASIC. What got me into computer club was there was a friendly face here, an empathic face. A face that I could ask, how does a computer work? I went to a parents’ evening where you could get the kids into extra-curricular subjects and you can stay behind after school and try certain things. I happened to talk to Mark and ask him, he was also in my school year, how does this computer work? He showed me. From that friendly face, this is why I’m talking to you now. This is proof beyond doubt that empathy does work, at least for me. Eventually, I persuaded my mum to get me an Acorn Electron. What I really wanted was the top right here, the BBC Micro Model B, but that cost £400. I ended up with the lower left, Acorn Electron. In the end, because the Acorn Electron is a cut-down BBC Micro, I was still able to program primitive games, learn assembly language, and write code like this that showed graphics. I honed my chops, just like a guitarist really, or a musician. I started coding. I really wanted to know more about it. I knew at 13 years old I was going to be something to do with computing. I didn’t know if it was going to be a software engineer or architect, I knew this was my vocation. I didn’t know how I was going to get there.

What I’m talking about here, I’m talking about the feelings for another, which is the seventh empathic capacity, which is, we express vulnerability when we’re learning. It takes somebody with compassion, and maybe even pity, and somebody who is formed enough to give us guidance. Mark could have told me to go just bugger off, or he could have been in a bad mood. Because he was there on that parent’s evening. Because I can imagine, I can play this game called perceptual positions, and imagine standing in Mark’s shoes, and a young black kid comes up to him and says, show me how this computer works. Maybe I was rude. I could have been good. I could have asked him. I cannot remember. I owe it to Mark because he set me on the road.

Why Are You Here?

Fast forward. This would have happened to you as well. If you perform and close your eyes, you will know how you found this mentor yourself. Once you always know, because you’re here watching this program, you can make it work for you always, just by using your cerebellum and imagining that you could get back to that moment when someone shows you your first code, your first project, your first laptop, your first ideas, or pointed you in a direction to become who you are or will be tomorrow.

Dealing with Restructuring

Let’s move on with a story. Graduated, ended up working in Germany. Came back to Britain in the mid-1990s just in time for Blur versus Oasis, discovered that. In fact, instead of working in the outskirts of London, the heart of the matter, or the technology was in central London in investment banking. At the same time, I knew about Bjarne Stroustrup and C++. I wanted to be just like the next Stan Lippman. I already became a Linux system admin here, almost did. Then, I happened to chance on to and stumble onto Java. I had stumbled onto Java a year before, but it didn’t click with me. When I joined banking, I said, this is Java? Then my life had changed. I was still interested in C++ a bit, but then Java definitely took over, by the millennium.

In the turn of millennium, Deutsche Bank, which is my first investment bank that I worked for. This is the office environment that I worked on, like hundreds of people with the old cathode ray tube monitors, maybe a few flat screens in this. This is before the financial bubble burst. Then, everything was hunky-dory, just like you’re using the best technology in the summer of the year 2000, and that lasted for another year to 2001. There was Java. There was all these Dotcoms, fabulous technologies. Java 2 was just about to come out. We still didn’t have broadband yet, if I remember, still had a bit of dialogue to get through. I was happy to hack on the bank’s computers there because they had the most up to date technologies there. It’s easy to be trapped in that environment. This is not less about empathy, but you can get trapped in a moment, the rewards of the environments that exist now, that means you can forget about tomorrow. September 11th happened, and the Dotcom bubble burst. I wish I was back at 1988. Life was a little bit simpler.

This is the biography continuing. Ten months later, my job was gone. There was Summer in the City. I became very lazy and very despondent. Ended up watching the football world cup. This is a guy called David Beckham, who’d take fabulous free kicks. Years later, I discovered agility and retrospectives, nearly 10 years later. If I could write or mark this on the board right then in the summer of 2001 and 2002 with a purple marker, this would be my positivity and how I felt. Because restructuring took it all out of me really. I was feeling really chipper, because I got money in compensation. Then everything trended down from there. You might wonder, what has this got to do with empathy? This is George Clooney and Kendrick, and if I’m Up in the Air, the movie, which does talk about restructuring and job loss. When you hit a downturn, you will become depressed, unless you’re one of these very lucky people who managed one week later to be rescued or find the next gig immediately. Banks then, especially Deutsche Bank, they paid for outplacement, a decent compensation, half a year’s salary, and help to kickstart a new job search and a new CV. A very good movie to watch, if you’re going through this.

Then, years later, I realized that everybody is sharing this shared pain, so feeling as another does in the shared circumstance. It’s birth, marriage, bereavement, war, famine, hunger, success, winning the world cup. Those are shared circumstances that you can share, that others can share with you. That is an empathy connection, which is useful in teams, for those of you who are managing, or thinking of managing. The first rule about restructuring, my advice, no matter, because it will happen to you. Your job is made redundant, you are never redundant. Say that again and say it well, your job is made redundant, you are never redundant. If you remember that phrase when it does eventually happen to you, because nobody ever escapes.

What do you have control of? You have control over you, your influence, and your reach. If you set fire to the school, with pupils in, you are influencing and harming students, as well as the teachers and the caretakers, so you have control and influence, and you have reach. The reach is to connect people to the wider community. It’s a very negative example, I know. Then let’s be Afrika Bambaataa about this in hip hop, turn that negative into a positive. That’s what I did with myself. I needed to feel restructuring. I needed that disappointment to lose the semifinal, or quarterfinal, if not the final itself in the tournament. Because I then asked questions about myself like a sportsperson. Where’s my community? What do I want to do? Should I stay with computing or leave it? I had empathetic thoughts. I was feeling these feelings but couldn’t coin a name for them, or couldn’t put my finger on to what I was feeling. Then, I happened to be contracting. I fell into contracting really. Then technology in the times changed, Java EE became big, Spring framework came out, financial services still had that rebound until 2008. Then it crashed again.

Imagining Another’s Thoughts and Feelings

Imagining another’s thoughts and feelings. I had dreams. What about other people in London, and also, working on technology? They had dreams, desires. I wanted to communicate. I wanted to learn and better myself. It just so happened that I finally did it. I didn’t destroy the world or blow up the world, I created this thing called the Java web user group, a user group because I asked a question, are there other independent engineers who are willing to travel with me to JavaOne 2004? It turns out not only I had the money because I was a contractor, but because I started this mini-community, I got good at the MPV parts of it as facilitating, connecting, organizing speakers, getting in contact with Wendy Devolder of Skills Matter. Making presentations, as well as at QCon, and eventually at JavaOne. Getting involved with other people in the wider community to do with Scala, and Groovy, Spring Boot, Hibernate, HTML5, and even JavaFX. There are 20 other technologies that I could list including OSGi, that I could throw in here. Eventually, in 2007, I became a Java champion. It’s because I was a contractor, I had money. I was able to travel and afford to see the greater and wider community that I became one myself. It’s not something that you force yourself or apply to join, other people have to look at your contributions to the wider Java community. This Java Champion program, that was created by Aaron Houston, once of Sun Microsystems, he was the global outreach for Sun Microsystems. He realized that this technology is just not a panacea. It is driven by the people who believe passionately that Java and technologies around Java can help in the world. That’s a huge lesson. He’s still out there somewhere.

At the same time, I was reading Malcolm Gladwell, about connectors are people you seem to know who seem to know everybody. You find connectors in every walk of like. They’re sociable, gregarious, naturally skilled at making and keeping in contact with friends and acquaintances. What Malcolm was talking about was the first genuine influencers, not the influencers that broadcast, but the influencers that facilitate, a facilitating influencer instead of taking money. In order to inspire interest, though, as a user group leader, and I tried to be as fair as I could. Be that connector between groups of diverse people and the audience. Remember to facilitate the conversation. I tried to do that and I did that well for much of my time as well in that user group.

Adopting Another’s Posture for Achievement

What I really did is seeing these disparate speakers and how they presented at not only the user group, the Java web user group, but also at conferences. I learned by watching Neal Gafter, and Joshua Bloch in Java, or Mark Reinhold, or Brian Goetz in Java, by adopting and learning and modeling their accents, and the way they attenuated their tone. They spoke. They informed. They were passionate. They drilled down. They were respectful. They had empathy. That’s who I wanted to be. I wanted to be like them. It’s like Gianfranco Zola, the Italian footballer. He was lucky enough to play with Diego Armando Maradona at Napoli. He always goes on. Zola always goes on, “I was so lucky to see Maradona train.” That’s a sporting, I suppose symbology works in technology too. If you can model excellence in your heroes and heroines, it is a good way to be. In order to do that, you need to understand empathy, which is what this talk is about.

Switching Careers

Hunky-dory came, you know what comes next. The huge financial crisis of Lehman Brothers. Eventually, contracting ended for me. I became full time. I survived Lehman Brothers financial meltdown. I knew many people at Lehman Brothers who were fantastic. On a Friday, sometime in September, everything’s hunky-dory, and over the whole weekend, a bank suddenly, an investment bank, Lehman Brothers suddenly gone. Incredible. This was a big meltdown. I don’t think we’ve had the boom time since 2008. Have you had boom times for the general population? Maybe Elon Musk had some boom times. We haven’t seen it, the ordinary folk in the trough. I stayed with investment banking. I tried up into 2012, I think, Olympic year. Then I’d had enough. Sometimes you do have to switch rails. My partner says, change it, leave it, or accept it. She is an NLP Master Practitioner. She has advised me time and time again, and other people have said the same thing. If you can’t change your organization then change the organization. In order to change the organization, you need empathy. That has to come from you. That’s the secret to that quote.

We get into the penultimate stage here. I left banking. This is the biographical part continuing, I became digital. That was the new way of earning money as a contractor. I stumbled into and I accepted digital web agency development. My head was blown. I heard about this user research and user centric web design, where you have call to action, the serious stuff, copywriting, content generation, structure of English, strategy, content strategy, all these new words. Here it is, including responsive website design. Suddenly, the frontend was taking off. Full stack hadn’t come quite in by 2013. I think full stack is a horrible term, it’s a lack of empathy, because people who are full stack, have a weakness. They may be good at frontend, but they may not be good at the backend, or in the Java side. I’ve never known someone to know all of the platform stack, including DevOps. It’s impossible. Computing subject matter is too big. If you want to get things done, you’re going to have to have empathy. You’ll find out next.

It is because at the same time, the governments were realizing that its digital websites were old, they were antiquated. They were broken, disparate. This lady here, Dame Martha Lane-Fox submitted a paper to Prime Minister David Cameron in 2010, to form gov.uk. This is basically to adopt agile and become saving space. Here, code is a communication vessel of the external intent and behavior, between two or more different human beings. Code itself had changed to be unmaintainable, to definitely maintainable. Somebody has to look at your code, and be able to compile it, or to transpile it, years from now. We have to be agile, because now things were changing.

I’m so ashamed that my first experience of XP was, I think, 2014, or 2015, where I actually paired with a lead developer who was better than I, in terms of the web, JavaScript side. I had to learn pairing with him and her. Then I stumbled into Santander and started programming, become a Spring Boot developer. What I discovered in the Santander project, which was 100% pairing in 2017, is that it really requires empathy. I didn’t have all the times then, but you have to be inclusive. I know this, people are racially inclusive. When you’re coding with a junior, that you have to allow juniors to make mistakes. Likewise, if you’re coding with someone better than you, then you are in the vulnerable state. The other thing about pairing, every two or three days, we would swap pairs, we would follow the Extreme or the Pivotal way of coding. No pair worked on the same story for two or three days, we will all swap a pair. Then we’d get knowledge sharing. That increases knowledge within the team. You’ll see that you quickly need to know each other’s thoughts and feelings for the win in order to get that agile performance and the team. That’s why people were using pair program, and finally I got it.

What I discovered is that in the industry, we do have this negative gamification, which is contempt. It’s a feeling of superiority over people, putting people down. What is required for high performance teams, and if you want to interview for a great team, you need compassion. It is required for effective agility. Then, when I dug deeper into the agile manifesto, one day, I was shocked to read that they’ve been talking about this in 2002. This type of situation goes on every day in marketing, management, external customers, internal customers, and yes, even developers don’t want to make hard tradeoff decisions, so they impose irrational demands for the imposition of corporate power structures. This isn’t merely about software development problem, it runs through Dilbertesque organization. Whoever wrote that as text was suffering because of lack of empathy. They were annoyed at what goes on in their organization. This is in 2001.

Interconnected, Empathetic Software Engineering

I joined that digital development. I had proper daily standups since January 2013. We are in 2020, and the world changed. I enjoyed my contracting days, but suddenly things changed. We had remote communication, remote repositories, remote deployment from being in remote laptops or positions, virtual private network issues. It was a whole smorgasbord, and then some. What’s my opinion of high-quality software engineering? A healthy codebase, systems and application looks like a model of communication between people who exude compassion, and empathy. For interconnected, empathetic software engineering, there are two sides to it. There’s the technical side, the maintainable side where we want less bit rot. We want testable code. We want adaptable code. We want code that is reusable, maybe, that is exchangeable, is that really required? Resilience? This is where we have the non-technical side. We want ethics, teamwork. We want to be open and seen. We want to be inclusive. We definitely need psychological safety, to ensure we also have fairness in the workplace. Because if people aren’t treated fairly, then you’re going to get attrition. We pair in a high-performance team, a high caliber team. Definitely, the people that I’ve worked with in those teams, they have synchronized and composed relationships. They get on. They have that trust and support. They collaborate. Sometimes that leads to innovation. If you want reusable fairness and innovation, get those things right.

Empathetic Toolset

What you need is this empathetic toolset, which first starts with actively listening. Then this is about the culture of content or addressing it. You definitely want to be actively listening, observing, and having that kinesthetic, that feeling. That is touching on the amygdala, listening. There’s three sides to every story. Everybody has strengths and weaknesses. We want to show that we comprehend others and other people. We want to avoid indignation. We want to show people are included. We can do that in the same way we date other people or interview. We mirror. We have rapport. We don’t want platitudes. We want to show that we are self-aware. We are sentient. We are nonjudgement. We also cultivate. We encourage. We collaborate. Then, if we are really lucky, and this is impossible, because even I have a big head, try to be egoless as much as possible. I know Ken Beck and the Scrum XP people proclaim this a lot.

High-Performance Teams

Evidence of high-performance teams. This comes from the Mind Gym, and less so about empathy, but really, is about it. Because the last bit of the sentence, the mixed gender groups consistently outperformed the single gender groups. This was a study about diversity, which is relevant. What’s even more relevant is Kelsey Hightower, who is a Kubernetes [inaudible 00:38:53], Developer Relations at Google. He talks about running a customer empathy session, where he puts Kubernetes engineers through the same challenges that his and her customers face. He’s still having fun. If you think you can avoid empathy, it’s there in the FAANG baby, in those FAANG companies, at least in one of them. It’s coming to a software developing engineering workshop and store that you are in right now. There is no escape, at least.

Empathic Capacities

Daniel Batson, who came up with the empathic capacities, American social psychologist, well known for the empathy-altruism hypothesis. Number one is knowing another’s thoughts and feelings. That is the feeling that we have almost when an event in a circumstance is happening, or when we think, just like the other person. Imagining another’s thoughts and feelings. This is slightly substantively different in that we are in a different plane. We are thinking of what it’s like to occupy that person’s corner of the universe. Adopting the posture of another person, the kind of method acting here. This is really useful for coaching. I believe this is true. Because if you want to learn from the best, then work with the best and model excellence. Feeling as another person does. This is the circumstances shared where you are in that same circumstance. You might be in Ukraine, in Mariupol right now, or luckily, you aren’t, you’re here with us, and you can definitely feel what is going on there.

Number five, imagining how one would feel or fit, or think in another’s place. Again, slightly related to number two and one, but this how is projecting into the future, this one here. If you cause somebody’s harm, or you put ice down that girl’s back, as you did as a child, what stops you doing that? That is not a nice feeling to do, because maybe that happened to you, someone squirted you with water, and you understand pleasure as well as pain. You can imagine how one would feel in another’s place. Feeling distress at another’s suffering. This is an easy one. Bereavement, somebody’s pet has died, family member receives medical bad news. Somebody has taken sick at work. The announcement of terminal cancer is a really severe one, and you can treat it with platitude. If you have a similar distress, or you know a family member that’s had that similar distress, you will know number six. Feeling for another’s suffering, which is the charitable way. That is more to do with pity and compassion. That means you’re not actively involved, but you can feel. It is closely related to five, really, I think. You can feel without actually taking aboard all their emotions. Number eight is the classic projecting oneself into another situation, the actor. Imagine yourself emoting, being in that same person’s shoes, as one says.

Conclusion

An inclusive environment is necessary to reap the benefits of employing a diverse group of people. I’m going to leave you with empathic leadership tips here. For those of you who are leaders, or want to become leaders, you want this seat right now that I’m sitting in, you definitely want to look at and study the things in yellow boxes, which is the blame culture, the unblocking of encumbrances, the sanctuary, and the flexible communication. I’m going to leave you with Bob Odenkirk from Breaking Bad, and Better Call Saul. He says you’re not owed a career just because you want one. How true is that? Empathy. We continuously practice empathy even when we don’t have a lot of it to start with. We share in everyone else’s lived-experiences in order to build that better future for all of us, so say we all.

See more presentations with transcripts

Article originally posted on InfoQ. Visit InfoQ

Transcript

Sameki: My name is Mehrnoosh Sameki. I’m a senior program manager and technical lead of the Responsible AI tooling team at Azure Machine Learning, Microsoft. I’m joining you to talk about operationalizing Responsible AI in practice. First, I would like to start by debunking the idea that responsible AI is an afterthought, or responsible AI is a nice-to-have. The reality is, we are all on a mission to make responsible AI the new AI. If you are putting an AI there, and it’s impacting people’s lives in a variety of different ways, you have the responsibility to ensure the world that you have created is not causing harm to humans. Responsible AI is a must-have.

Machine Learning in Real Life

The reason why it matters a lot is, besides the very fact that in a traditional machine learning lifecycle, you have data, then you pass it to a learning phase where that learning algorithm takes out patterns from your data. Then that creates a model entity for you. Then you use techniques like statistical cross-validation, accuracy, to validate and evaluate that model and improve it, leading to a model that then spits out information such as approval versus rejection for loan scenarios. Despite the fact that this lifecycle matters, and you want to make sure that you’re doing it as reliably as you could, there are lots of human personas in the loop that need to be informed in every single stage of this lifecycle. One persona, or many of you, ML professionals, or data scientists, they would like to know what is happening in their AI systems, because they would like to understand if their model is any good, whether they can improve their model, what features of their models should they use in order to make reliable decisions for humans? The other persona are business or product leaders. Those are people who would like to approve the model, should we put it out there? Is it going to put us on the first page of the news another day? They ask a lot of questions from data scientists regarding, is this model racist? Is this biased? Should I let it be deployed? Are these predictions matching some domain experts’ insights that I’ve got from surgeons, doctors, financial experts, insurance experts?

The other persona is end-users, or solution providers. By that I mean either the banking person who works at a bank and is providing people with that end result of approved versus rejected on their loan, or a doctor who is looking at the AI results and is providing some diagnosis or insights to the end user or patient, in this case. Those are people who deal with the end user. Or they might be the end user themselves. They might ask, why did the model say this about me, or about my patient or my client? Can I trust these predictions? Can I make some actionable movements based on that or not? One persona that I’m not showing here, but is overseeing the whole process, are the regulators. We all have heard about the recent European regulations and GDPR’s right to explanation, or California act. They’re all adding lots of great lenses to the whole equation. There are risk officers, regulators who want to make sure that your AI is following the regulations as it should.

Microsoft’s AI Principles

With all of these great personas in the loop, it is important to ensure that your AI is being developed and deployed responsibly. However, even if you are a systematic data scientist or machine learning developer and really care about this area, truth to be told is the path to deploying responsible and reliable machine learning is still unpaved. Often, I see people using lots of different fragmented tools, or a spaghetti of visualizations or visualization primitives together in order to evaluate their models responsibly. That’s our team mission to help you operationalize responsible AI in practice. Microsoft have these six principles in order to inform your AI development and deployment. Those are fairness, reliability and safety, privacy and security, inclusiveness, underpinned by two more foundational ones: transparency and accountability. Our team specifically works on the items that are shown in blue, which are fairness, reliability and safety, inclusiveness, and transparency. The reason why we work on them is because they have a theme. All of these are supposed to help you understand your model better, whether through the lens of fairness or through the lens of how it’s making its prediction, or through the lens of reliability and safety and its errors, or whether it’s inclusive to everyone. Hopefully, help you build trust, improve it, debug it further, and make actionable insights.

Azure Machine Learning – Responsible AI Tools

Let’s go through that ecosystem. In order to guide you through this set of tools, I would like to first start by a framing. Whenever you are having a machine learning lifecycle, or even just data, you would like to go through this cycle. First, you would like to take your model and identify all the issues, aka, fairness issues, errors, that are happening inside that. Without identification stage, you don’t know exactly what is going wrong. Next, another important step is to diagnose why that thing is going wrong. The diagnosis piece might look like that, now I understand that there are some issues or errors in my data. Now I diagnose that the imbalance in my data is causing it. The diagnosis stage is quite important, because that discovers the root cause of the issue. That’s how you can take more efficient, targeted mitigations in order to improve your model. Naturally, then you move to the mitigation stage where, thanks to your identification and diagnosis skills, now you can mitigate those issues that are happening. One last step that I would like to highlight is take action, sometimes you would like to inform a customer or a patient or a financial loan applicant about, for instance, what can they do, so next time they get a better outcome. Or, you want to inform your business stakeholders as what can you give some of the clients in order to boost sales. Sometimes you want to take real-world actions, some of them are model driven, some of them are data driven.

Identification Phase

Let’s start with identify and the set of open source tools and Azure ML integrated tools that we provide for you to identify your model issues. Those two tools are error analysis and fairness tools. First, starting with error analysis, the whole motivation behind us putting this tool out there is the fact that we see people often use one metric to talk about their model’s goodness, like they say, my model is 73% accurate. While that is a great proxy into identifying the model goodness and model health, it often hides this important information, that error is not uniformly distributed in your data. There might be the case that there are some erroneous packets of data, like this packet of data that is only 42% accurate. Versus, in contrast, this packet of data is getting all of the right predictions. If you go with one number, you’re losing this very important information that my model has some erroneous packets, and I need to investigate why that cohort is getting more errors. We released a toolkit called error analysis, which is helping you to validate different cohorts, understand and observe how the error has been distributed across your dataset, and basically see a heat map of your errors as well.

Next, we worked on another tool called Fairlearn, which is also open source, it is to help you understand your model fairness issues. It is focusing on two different types of harms that AI often give rise to. One is harm of quality of service, where AI is providing different quality of service to different groups of people. The other one is harm of allocation where AI is allocating information opportunities or resources differently across different groups of people. An example for harm of quality of service is a voice detection system that might not work as well for say females versus males or non-binary people. An example of harm of allocation is a loan allocation AI or a job screening AI that might be better at picking candidates among white men compared to other groups. The whole hope behind our tool is to ensure that you are looking at the fairness metrics with the lens of group fairness, so how different groups of people are getting this treatment. We provide a variety of different fairness and performance metrics and rich visualizations, in order for you to observe the fairness issues as they occur in your model.

Both of these support a variety of different model formats, Python model using scikit predict convention, Scikit, or TensorFlow, PyTorch, Keras models. They also support both classification and regression. An example of a company putting our fairness tool into production is Philips Healthcare. They put fairness in production into their ICU models. They wanted to make sure that their ICU models that they have out there is performing uniformly across different patients with different ethnicities, gender identities. Another example is Ernst & Young in a financial scenario where they use this tool in order to understand how their loan allocation AI is providing this opportunity of getting a loan across different genders and different ethnicities. They were able to also use our mitigation techniques.

Diagnosis Phase

After the identification phase, now you know where the errors are occurring, and you know your fairness issues. You move on to the diagnosis piece. I cover two of the most important diagnosis capabilities, interpretability and perturbations and counterfactuals. One more to just like momentarily touch on is, we’re also in the process of releasing a data exploration and data mitigation library. The diagnosis piece right now entails the more basic data explorer. I will show that to you in a demo. It also includes interpretability, that’s the module we provide to you, which basically tells you what are the top key important factors impacting your model predictions. How your model is making its predictions. It covers both global explanation and local explanation. How overall the model is making its prediction, and how individual data points for them, how the model has made its predictions.

We do have different packages under Interpret ML capabilities that we have. It’s a collection of black box interpretability techniques that can literally cover any model that you bring to us, no matter if it’s Python, or Scikit, or TensorFlow, PyTorch, Keras. We also have a collection of glassbox models that are intrinsically interpretable models, if you have the flexibility of basically changing your model and training an interpretable model from scratch. An example of that is Scandinavian Airlines. They basically used our interpretability capabilities via Azure Machine Learning to build trust with their fraud detection model of their loyalty program. Of course, you can imagine that in such cases, you want to reduce and minimize and remove mistakes, because you don’t want to tell a very loyal customer that they’ve done some fraudulent activity, or flag their activity by mistake. That is a very bad customer experience. They wanted to understand how their fraud detection model is making their predictions, and so they used interpretability capabilities to understand that.

Another important diagnosis piece is counterfactual and perturbations. You can do lots of freeform perturbations, do what-if analysis, change features of a data point, and see how the model predictions change for that. Also, you can look at counterfactuals and that is simply telling you what is the bare minimum changes to a data point’s feature values that could lead into a different prediction. Say, Mehrnoosh’s loan is getting rejected, what is the bare minimum change that I can apply to her features so that the AI predicts approved next time?

Mitigate, and Take Action Phase

Finally, we go to the mitigation stage, and also take action stage. We do cover a class of unfairness mitigation algorithms that could literally encompass any model. They have different flexibilities. Some of them are just post-processing methods and could adjust your model predictions in order to improve it. Some of them are more like reductions method, combination of pre-processing and in-processing. They can update your model objective function in order to retrain your model and not just minimize error, but also put control on a fairness criteria that you specify. We also do have pre-processing methods that will readjust your data in terms of better balancing it and better representing the underrepresented groups. Then, hopefully, the model that is trained on that augmented data is going to be a fairer model. Last, we realized that a lot of people are using our model, Responsible AI insights, for decision making in the real world. We all know models sometimes take on correlations rather than causation. We wanted to provide you with a tool that works on your data, just historic data, and uses a technique called double machine learning in order to understand whether there are any causal effects of a certain feature on the real-world phenomenon. Say, if I provide promotion to a customer, would that really increase the sales that that customer will generate for me? Causal inference is another capability we just released.

Looking forward, one thing that I want to mention is, while I went through different parts of this identify, diagnose, mitigate, we have brought every single tool I just represented under one roof, and that is called Responsible AI dashboard. The Responsible AI dashboard is a single pane of glass, bringing together a variety of these tools under one roof, same set of API, a customizable dashboard. You can do both model debugging and also responsible decision making with that, depending on how you’re customizing and what you pass to it. Our next steps would be to expand the portfolio of Responsible AI tools to non-tabular data, enable Responsible AI reports for non-technical stakeholders. We do have some exciting work on PDF reports you can share with your regulators, risk officers, business stakeholders. We are working on enabling model monitoring at scoring time just to bring all these capabilities beyond evaluation time and bring it to scoring time, and make sure that as the model is seeing the unseen data, it can still detect some of these fairness issues, reliability issues, interpretability issues. We’re also working on a compliance infrastructure, because we all know that there are nowadays so many stakeholders involved in development, deployment, and testing and approval of an AI system. We want to provide the whole ecosystem to you.

Demo

We believe in the potential of AI for improving and transforming our lives. We also know there is a need for tools to assist data scientists, developers, and decision makers to understand and improve their models to ensure AI is benefiting all of us. That’s why we have created a variety of tools to help operationalize Responsible AI in practice. Data scientists tend to use these tools together in order to holistically evaluate their models. We are now introducing the Responsible AI dashboard, which is a single pane of glass, bringing together a number of Responsible AI tools. With this dashboard, you can identify model errors, diagnose why those errors are happening, and mitigate them. Then, provide actionable insights to your stakeholders and customers. Let’s see this in action.

First, I have here a machine learning model that can predict whether a house will sell for more than median price or not, and provide the seller with some advice on how best to price it. Of course, I would like to avoid underestimating the actual price as an inaccurate price could impact seller profits and the ability to access finance from a bank. I turned into the Responsible AI dashboard to look closely at this model. Here is the dashboard. I can do, first, error analysis to find issues in my model. You can see it has automatically separated the cohorts with error counts. I found out that bigger old houses have a much higher error rate of 25% almost in comparison with large new houses that have error rates of only 6%. This is an issue. Let’s investigate that further. First, let me save these two cohorts. I save them as new and old houses, and I go to the model statistics for further exploration. I can take a look at the accuracy, false positive rates, false negative rate across these two different cohorts. I can also observe the prediction probability distribution and observe that older houses have higher probability of getting predictions less than median. I can further go to the Data Explorer and explore the ground truth values behind those cohorts. Let me set that to look at the ground truth values. First, I will start from my new houses cohort. As you can see here, most of the newer homes sell for higher price than median. It’s easy for the model to predict that and get a higher accuracy for that. If I switch to the older houses, as you can see, I don’t have enough data representing expensive old houses. One possible action for me is to collect more of this data and retrain the model.

Let’s now look at the model explanations and understand how the model has made its predictions. I can see that the overall finish quality, above ground living room area, and total basement square footage are the top three important factors that impact my model’s prediction. I can further click on any of these like overall finish quality, and understand that a lower finish quality impacts the price prediction negatively. This is a great sanity check that the model is doing the right thing. I can further go to the individual feature importance, click on one or a handful of data points and see how the model has made predictions for them. Further, when I come to the what-if counterfactual, what I am seeing here is for any of these houses, I can understand what is the minimum change I can apply to, for instance, this particular house? Which has actually a high probability of getting the prediction of less than median, so that the model predicts the opposite outcome. Looking at the counterfactuals for this one, only if the house had a higher overall quality from 6 to 10, then the model would predict that this house would sell for more than median. To conclude, I learned that my model is making predictions based on the factors that made sense to me as an expert, and I need to augment my data on the expensive old house category, and even potentially bring in more descriptive features that help the model learn about an expensive old house.

Now that we understood the model better, let’s provide house owners with insights as to what to improve in these houses to get a better price ask in the market. We only need some historic data of the housing market to do so. Now I go to the causal inference capabilities of this dashboard to achieve that. There are two different functionalities that could be quite helpful here. First, the aggregate causal effect which shows how changing a particular factor like garages, or fireplaces, or overall condition would impact the overall house price in this dataset on average. I can further go to the treatment policy to see the best future intervention, say switching it to screen porch. For instance, here I can see for some houses, if I want to invest in transforming a screen porch, for some houses, I need to shrink it or remove it. For some houses, it’s recommending me to expand on it. Finally, there’s also an individual causal effect capability that tells me how this works for a particular data point. This is a certain house. First, I can see how each factor would impact the actual price of the house in the market. I can even do causal what-if analysis, which is something like if I change the overall condition to a higher value, what boost I’m going to see in the housing price of this in the market.

Summary

We looked at how these tools help you identify and diagnose error in a house price prediction model and make effective data-driven decisions. Imagine if this was a model that predicted the cost of healthcare procedures or a model to detect potential money laundering behavior, identifying, diagnosing, or making effective data-driven decisions would have even higher consequences on people’s lives there. Learn more about the tool on aka.ms/responsibleaidashboard, and try it on Azure Machine Learning to boost trust in your AI driven solutions.

Questions and Answers

Breviu: Ethics in AI is something I’m very passionate about. There’s so much harm that can be done if it’s not thought about. I think that showing the different tools and the different kinds of thought processes that you have to go through in order to make sure that you’re making models that are going to not only predict well for accuracy, but also that they’re not going to cause harm.

Sameki: That is absolutely true. I feel like the technology is not going to slow down. We’re just starting with AI and we’re expanding on its capabilities and including it in more aspects of our lives, from financial scenarios, to healthcare scenarios, to even retail, our shopping experience and everything. It’s even more important to have technology that is accompanying that fast growth of AI and is taking care of all those harms in terms of understanding them, providing solutions or mitigations to them. I’m quite excited to build on these tools and help different companies operationalize this super complicated buzzword in practice, really.

Breviu: That’s true. In so many companies, they might want to do it, but they don’t really know how. I think it’s cool that you showed some of the different tools that are out there. There was that short link that you provided that was to go look at some of the different tools. You also mentioned some new tooling that is coming out, some data tooling.

Sameki: There are a couple of capabilities. One is, we completely realized that the model story is incomplete without the right data tools, or data story. Data is always a huge part, probably the most important part of a machine learning lifecycle. We are also accompanying this with more sophisticated data exploration and data mitigation library, which is going to land under the same Responsible AI toolbox. That will help you understand your data balances, and that also provides lots of APIs that can rebalance and resample parts of your data that are underrepresented. Besides this, at Microsoft Build, we’re going to release a variety of different capabilities of this dashboard integrated inside our Azure Machine Learning. If your team is on Azure Machine Learning, you will get easy access, not just to this Responsible AI dashboard and its platform, but also a scorecard, which is a report PDF, summarizing the insights of this dashboard for non-technical stakeholders. It was quite important for us to also work on that scorecard because there are tons of stakeholders involved in an end-to-end ML lifecycle. Many of those are not super data science savvy or super technical. There might be surgeons. There might be financial experts. There might be business managers. It was quite important for us to also create that scorecard to bridge the gap between super technical stakeholders and non-technical stakeholders in an ML lifecycle.

Breviu: That’s a really good point. You have the people that understand the data and how to build the model, but they might not understand the business application side of it. You have all these different people that need to be able to communicate and understand how their model is being understood. It’s cool that these tools can do that.

You talked about imbalanced data as well. What are some of the main contributing factors to ethical issues within models?

Sameki: Definitely, imbalanced data is one of them. That could mean many different things. You are completely underrepresenting a certain group in your data, or you are representing that group, but that group in the training data is associated with unfavorable outcomes. For instance, you have a certain ethnicity in your loan allocation AI dataset, however, all of the data points that you have from that ethnicity happen to have rejection on their loans. The model creates that association between that rejection and belonging to that ethnicity. Either not representing a certain group at all, or representing them but not checking whether they are represented well in terms of the outcome that is affiliated with them.

There are some other interesting things as well, after the data, which is probably the most important issue. Then there is the issue of problem definition. Sometimes you’re rushing to train a machine learning model on a problem, and so you’re using the wrong proxies as a predictor for something else. To give you a tangible example, to make it understandable, imagine you do have a particular model that you’re training in order to assign different risk scores to neighborhoods, like security scores. Then you realize that, how is that model trained? That model is trained on a data that is coming from arrest records of the police, imagine. Just using arrest records as a proxy into the security score of a neighborhood is a very wrong assumption to make because we all know that policing practices at least in the U.S. is quite unfair. It might be the case that there are more police officers deployed to certain areas that have certain ethnicities, and way less police officers to some other areas where there are some other ethnicities residing. Just because there are more police officers there, there might be more reporting of certain even like misdemeanors, or something that that police officer didn’t like, or whatever. That will bump up the number of arrest records. Using that purely for proxying to the safety score of that neighborhood, has that dangerous outcome of affiliation between the certain race residing in that neighborhood and the security of that neighborhood.

Breviu: When those kinds of questions come up, I think about, are we building a model that even should be built? Because there’s two kinds of questions when it comes to ethics in AI. It’s, is my model ethical? Then there’s the opposite, is it ethical to build my model? When you’re talking about arrest records, and that kind of thing, and using that, I start worrying about, what is that model going to actually do? What are they going to use that model for? Is there even a fair way to build the model on that type of data?

Sameki: I absolutely agree. A while ago, there was this project from Stanford, it was called Gaydar. It was a project, which was training machine learning models on top of bunch of photos that they had recorded and captured from the internet and from different public datasets. The outcome was to predict whether the person is belonging to the LGBTQ community or not, or gay or not. At that time, when I saw that I was like, who is supposed to use this and for what reason? I think that started getting a lot of attention in the media that, we know that maybe AI could do things like that, questionable, but maybe. What is the point of this model? Who is going to use it? How are we going to guarantee that this model is not going to be used to basically perpetuate biases, stuff like that, against the LGBTQ community that are historically marginalized? There are tons of deep questions that we have to ask that whether machine learning is an appropriate thing to do for a problem, and what type of consequences it could have. If we do have a legit case for AI, could be helpful to make processes more efficient, could be more helpful to expedite certain super lengthy processes. Then we have to accompany it with enough checks and balances, scorecards, and also terms and services as how people use that model. Make sure that we do have a means of hearing other people’s feedback in case they observe this model being misused in bad scenarios.

Breviu: That’s a really good example of one that just shouldn’t have happened. It always tends to be the marginalized, or the oppressed society, or parts of society that are hurt the most, and oftentimes aren’t necessarily the ones that are even involved in building it as well, which is one of the reasons why having a diverse set of engineering for these types of models. Because I guarantee you, if you had somebody that was part of that community building that model, they probably would have said, this is really offensive.

Sameki: They would catch it. I always knew about the focus of the companies on the concept of diversity and inclusion before I joined this Responsible AI effort, but now I understand it from a different point of view that, it matters, that we have representation from people who are impacted by that AI in the room to be able to catch these harms. This is an area where growth mindset is the most important. I am quite sure that even if we are systematic engineers that truly care about this area and put all of these checks and balances, stuff happens still. Because this is a very sociotechnical area where we cannot fully claim that we are debiasing a model. This is a concept that has been studied by philosophers and social scientists for centuries. We can’t come up suddenly out of the tech world and say, we’ve found a solution for it. I think progress could be made to figure out these harms, catching it early on, diagnosing why those happen. Mitigating them based on your knowledge, and documenting what you could not resolve and put some diverse groups of people in the decision making to catch some of those mistakes. Then, have a very beautiful feedback loop where you capture some thoughts from the audience and you are able to act fast and also very solid monitoring lifecycle.

Breviu: That’s actually a good point, because it’s not only just the ideation of it, should I do this? Ok, I should. Now I’m building it, now, make sure that it’s ethical. Then there’s the data drift and models getting stale and needing to monitor what’s happening in [inaudible 00:35:32], so make sure that it continues to be able to predict well, and do that.

Any of these AI tools that you’ve been showing, are they able to be used in a monitoring format as well?

Sameki: Yes. Most of these tools could be, for instance, the interpretability. We do have support of scoring time interpretability, which basically allows you to call the deployed model, get the model predictions, and then call the deployed explainer and get the model explanations for that prediction at runtime. The fairness error analysis pieces are a little trickier. Fairness, basically, you can also specify the favorable outcome, and you can keep monitoring that favorable outcome distribution across different ethnicities, different genders, different sensitive groups, whatever that means to you. For the rest of fairness metrics, or error analysis, and things like that, you might require, periodically upload some labeled data based on your new data, take a piece, maybe use crowdsourcing or human labelers to label that and then parse it. General answer is yes. There are some caveats. We’re also working on a very strong monitoring story that goes around these caveats and helps you monitor that during runtime.

Breviu: Another example, I think of ones where I’ve seen that make me uncomfortable, and this happens, like machine learning models as part of the interview process. It’s one that actually happens a lot. There’s already so many microaggressions and unconscious biases, that using a model like this in the interview process, and I’ve read so many stories about it as well, where having, just even on resumes, how quickly it actually is biased. How do you feel about that particular type of use case? Do you think these tools can work on that type of problem? Do you think we could solve it enough to where it would be ethical to use it in the interviewing process?

Sameki: I have seen both with some external companies, they’re using AI in candidate screening, and they have been interested in using the Responsible AI tools. LinkedIn is now also part of Microsoft family. I know LinkedIn is also very careful about how these models are trained, tested. I actually think these models could be great initial proxies to figure out some better candidates. However, it’s quite important that if you want to trust the top ranked candidates, it’s super important to understand how the model has picked that, and so look at the model explainability, because often, there has been this case of associations.

There are two examples that I can give you. I remember that once there was this public case study from LinkedIn, they had trained a model for job recommendations, how you go to LinkedIn and it says, apply for this and this. Then they realized early on that one of the ways that LinkedIn algorithm was using the profiles in order to match them with the job opportunities was the fact that the person was providing enough description about what they are doing, what are they passionate about? How you have a bio section and then you have your current position, which you can add text to. Then there was a follow-up study by LinkedIn which was mentioning that women tend to have less details shared there, so in a way women tend to market themselves in a less savvy way compared to men. That’s why men were getting better quality recommendations and a lot more matches compared to women or females, non-male, basically: females, non-binary. That was a very great wake-up call for LinkedIn, that, ok, this algorithm is doing this matching, we have to change it in order to not put too much emphasis. It’s great that they have this extra commentary and whatever. First of all, we have to maybe provide some recommendations to people who have not filled those sections as your profile is this much complete, how they give you signals as go and add more context.

Also, we have to revisit our algorithms to really look at the bare minimum stuff, like the latest position posted, experiences. Even then, women go on maternity leave and family care leaves all the time. I still feel like when we have these companies receiving so many candidates and resumes, there is some role that AI could play to bring some candidates up. However, before deploying it in production, we have to look at the examples. We have to also have a little team of diverse stakeholders in the loop to get those predictions and try to take a look at that from the point of view of diversity and inclusion, from the point of view of explainability of the AI, and interfere with some human rules in order to make sure it’s not unfair to some underrepresented candidates.

Breviu: That talks to the interesting thing, I think that you said, one of the beginning things is how the errors are not evenly distributed throughout the data. This is an example where, your model might get a really great accuracy but it was looking at the holistic approach, and realizing that on the non-male, female non-binary side that it was at a very high error rate. That’s like a really good example of that point that you made in the beginning, which I found really interesting. Because many times when we’re building these models, we’re looking at our overall accuracy rate and our validation and loss score. Those are looking at it as a holistic thing, not necessarily on an individual basis.

Sameki: It’s very interesting, because many people use platforms like Kaggle to learn about applied machine learning. Even in those platforms, we often see scoreboards where one factor is used to pick the winner, like accuracy of the model, area under curve, whatever that might be. That implicitly gives out that impression that, ok, there are a couple of proxies, if it’s good, great, go ahead and deploy. I think that’s the mindset that we would love to change in the market through this type of presentations that it’s great to look at your model goodness, accuracy, false positive rate, all those metrics that we’re familiar with for different types of problems. However, they’re not sufficient to tell you about the nuances of how that model is truly impacting the underrepresented groups. Or any blind spots, they’re not going to give you the blind spots. It’s not even always about fairness. Imagine, you realize that your model is 89% accurate or 95% accurate, but you realize that those 5% errors happen to happen for every single time we have this autonomous car AI, and the weather is foggy and dark and is rainy, and the pedestrian is a darker skin tone wearing dark clothes. Ninety-nine percent of the time the pedestrian is missed. That’s a huge safety and reliability issue that your model has. That’s a huge blind spot that is potentially killing people. If you go with one score about the goodness of the model, you’re missing that important information that your model has these blind spots.

Breviu: I think your point about Kaggle too in the ethics thing, kind of just shows where this ethics was an afterthought in a lot of this, and that’s why these popular platforms don’t really necessarily have those tools built in, as Azure Machine Learning does. I think also as we progress and people realize more just about like data privacy as well, I think as data scientists, we’ve always understood the importance of data privacy. I think now it’s becoming more mainstream. I think that part, and then understanding ethics more, I think it really will change how and the way that people build models and think about building models. I think AI is going to keep moving forward exponentially, in my opinion. It needs to move forward in an ethical, fully thought out way.

Sameki: We build all of these tools in the open source first, to help everyone explore these tools, augment it with us, build on it, and bring their own capabilities and components, and put it inside that Responsible AI dashboard. If you’re interested, check out our open source offering and send us a GitHub issue, send us your request. We are quite active on GitHub, and we’d love to hear your thoughts.

See more presentations with transcripts

Article originally posted on InfoQ. Visit InfoQ

Subscribe on:

Apple Podcasts
Google Podcasts
Soundcloud
Spotify
Overcast
Podcast Feed

.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.

Article originally posted on InfoQ. Visit InfoQ

Microsoft recently announced the preview of Azure Cosmos DB for MongoDB vCore, a fully managed MongoDB-compatible database service that provides developers with a familiar architecture for building modern applications with native Azure integrations and a low total cost of ownership.

With the additional Azure Cosmos DB for MongoDB vCore, developers can now choose between two architectures for running their data workloads on Azure Cosmos DB – the original throughput Request Unit (RU) and the newly designed vCore architecture.

Source: https://devblogs.microsoft.com/cosmosdb/azure-cosmos-db-for-mongodb-vcore-helps-developers-build-intelligent-apps-in-the-cloud/

According to the Microsoft documentation, the RU-based option works well for building new cloud-native MongoDB apps or refactoring existing apps. In contrast, the vCore-based option is better suited for lifting and shifting existing MongoDB apps to run as-is on a fully supported managed service. In addition, the RU-based option offers up to 99.999% availability with multi-region deployments and limitless horizontal scalability, while the vCore-based option offers competitive SLA (once generally available) and high-capacity vertical and horizontal scaling with vCore-based cluster tier options.

Furthermore, the RU-based option is better suited for workloads with more point reads, fewer long-running queries, and complex aggregation pipeline operations. On the other hand, the vCore-based option can handle workloads with any operation types, including long-running queries, complex aggregation pipelines, distributed transactions, joins, and so on.

Developers can create an Azure Cosmos DB for MongoDB vCore cluster that includes all the MongoDB resources (databases, collections, and documents) and a unique endpoint for various tools and SDKs for connectivity by using the Azure portal or bicep template. Furthermore, some everyday use cases can be storing and querying user-generated content (UGC) for web, mobile, and social media applications, such as chat sessions, tweets, blog posts, ratings, and comments.

Nikisha Reyes-Grange, Director of Product Marketing, Azure Data, told InfoQ:

Empowering MongoDB developers and removing the friction they encounter in the development process of intelligent applications has never been more important. Azure Cosmos DB for MongoDB vCore offers integrations with many Azure services, a familiar vCore architecture, predictable pricing, and enhanced querying to support apps such as chatbots.

Lastly, there are variable fees for RUs and consumed storage with the RU-based option, while with the vCore-based option, customers will pay for the number of vCores provisioned. More details on pricing and availability are available on the pricing page.

Article originally posted on InfoQ. Visit InfoQ

Microsoft has open sourced Semantic Kernel (SK), a lightweight SDK enabling the integration of large language models (LLMs) with conventional programs which can leverage prompt templating, vectorized memory, intelligent planning, and other capabilities.

Semantic Kernel is designed to support and encapsulate several design patterns from the latest in AI research, such that developers can infuse their applications with complex skills like prompt chaining, recursive reasoning, summarization, zero/few-shot learning, contextual memory, long-term memory, embeddings, semantic indexing, planning, and accessing external knowledge stores as well as your own data.

Prompt templating is a mechanism used to define and compose AI functions using plain text. To build prompts, developers can embed expressions within prompts using curly braces. For example, this is how you can call an external function passing an argument to it:

The weather today in {{$city}} is {{weather.getForecast $city}}.

Embeddings make it possible to map each word in a vocabulary to a point in a high-dimensional space that represents its meaning and the relationships between words. This is essential to solutions such as sentiment analysis, document classification, and recommendation systems.

In SK, embeddings are used to create a semantic memory, which mimics how the human brain stores and retrieves knowledge about the world based on the distance existing between concepts. Developers can either use pre-trained embedding models, or train their own using methods such as Word2Vec, GloVe, and FastText.

Another key capability SK provides out of the box is intelligent planning, which consists in working backwards from a user-provided goal. The planner builds thus a reverse sequence of steps, at each step leveraging a pre-defined skill available in a skill library. In case a required skill is not present, the planner can suggest a developer to create that skill or even help them writing the missing skill.

At a lower level of abstraction, a skill is simply a container for a function or a set of functions. In SK, there are three categories of skills: core skills, semantic skills (LLM prompts), and native skills, which are implemented by native code. Microsoft is also providing a number of sample skills for developers to learn how to create one, including chat, coding, classification, summarizing skills and others.

If you are interested to try out Microsoft Semantic Kernel, a good place to start is any of the sample apps that it includes as well as the official Getting Started notebook that will introduce you to the basic syntax to create semantic functions, embeddings, and so on.

Article originally posted on InfoQ. Visit InfoQ

Asking people how they feel about taking certain types of risks can give insight into the level of psychological safety and help uncover issues. Discussing the answers can strengthen the level of safety of more mature teams and help less mature teams to understand how they could improve.

Jitesh Gosai shared his experience with psychological safety in his talk at Lean Agile Scotland 2022.

Measuring psychological safety levels in teams is challenging as it’s hard to judge if someone feels like they are taking an interpersonal risk. Most people don’t think about risk-taking this way, and usually take a chance based on their situation and feelings, Gosai said.

One way to do it is to use a proxy measure of how people think about taking certain types of risks and if they feel supported to do so in their team:

While this still doesn’t give you a definitive scale of psychological safety in a team, or even if people will take interpersonal risks, it can at least indicate if you have any issues that would prevent people from doing so.

Using this scale is a very simple mechanism for understanding where a team is with psychological safety, Gosai mentioned. How people interpret questions can vary and how they respond can depend on how they feel that day.

Gosai mentioned two risks that could come from trying to measure psychological safety in teams. One is that if the responses are not anonymous, an individual could be singled out for pulling down the team average. This could make them feel they are being scrutinised for speaking up, potentially making them less likely to do so in the future. Another consequence of this could be that if others see this happening, they too may hold back, Gosai said.

The other risk Gosai mentioned is if these ratings are used to compare teams and their leads, rather than used to offer help to teams. Then they become just another metric to try and game, giving an even more skewed view of the team.

Gosai mentioned that most teams, instead of trying to measure psychological safety, would benefit from just starting to build a shared understanding of what it is first:

It could seem wasteful for teams with high levels, but they still stand to benefit by strengthening what they have. In contrast, those teams with low levels have everything to gain.

InfoQ interviewed Jitesh Gosai about measuring psychological safety.

InfoQ: How do you measure psychological safety?

Jitesh Gosai: Amy Edmundson provided a set of sample questions in her book The Fearless Organisation that team members can answer using a seven-point Likert scale which can be used as a starting point to gauge where teams are with psychological safety.

Based on how they respond, leaders can see if they have low, medium or high levels of psychological safety. For instance, if they score on the low end of the scale, then that could be seen as low levels of psychological safety, and leaders should take immediate action.

InfoQ: What did you learn from measuring the level of psychological safety?

Gosai: An ideal situation would be that questions related to psychological safety are incorporated into regular team health check questionnaires so that teams can see if there is an overall trend.

If they are trending down, then the team can look to see what has happened recently to cause this, but also, if they are trending up, they can see what they improved to do so. This way, the team can see the benefit of answering the questions as accurately as possible, rather than just another tick-box exercise.

When working with leaders, I find they know how their behaviours affect teams, but they often haven’t made that link to psychological safety. So helping them make that connection could immediately impact psychological safety in teams.

In Learnings from Applying Psychological Safety Across Teams, Gosai explained how they applied ideas from psychological safety to make it safe for people to take interpersonal risks. Creating Environments High in Psychological Safety with a Combined Top-Down and Bottom-Up Approach describes how complementing leadership with team workshops in communication skills can enable people to speak up and feel safe to fail.

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

Since Oracle launched its MySQL HeatWave service in December 2020, it has continuously driven differentiation in the database-as-a-service market. While competing against some of the biggest names in the cloud space, the company has shown what appears to be “too good to be true” price-for-performance against competitors such as Snowflake.

With the latest update to MySQL HeatWave, Oracle shows no signs of slowing down. And I think this service may be the “killer app” for Oracle Cloud Infrastructure (OCI)—or even Oracle as a whole. I’ll explain my reasoning in this article.

MySQL HeatWave changed the analytics game

Most IT folks who have spent time around databases understand the pervasiveness of MySQL in the enterprise. Since it was released under the GNU General Public License (GPL) in 2000, the database platform has exploded in popularity among companies of all sizes, from smaller organizations that want a SQL database without the considerable cost, to enterprise organizations deploying large departmental databases. My first exposure to the platform came in 2003, when I worked as an IT executive in a large state government organization. We used MySQL everywhere possible to lower costs without sacrificing performance and functionality.

Over the years, MySQL became the world’s most popular open-source database and the second-most popular database overall (after Oracle Database). Yes, while a lot of the buzz is for the likes of MongoDB and other “cool” NoSQL platforms, the top two database distributions—by far—are owned by Oracle. And they are both SQL-based.

The challenge I saw when running a state IT organization is the same challenge that IT orgs have been facing ever since: data silos and the fact that MySQL is not optimized for analytics. When a business has hundreds or even thousands of MySQL instances, integrating all of that data and gleaning insights from it via analytics is painful. All too often, business users must rely on IT to perform time-consuming and costly—yet error-prone! —extract, load and transform (ETL) processes to bring all the data to one central location for analysis. Or brave business users might attempt this themselves, then reach out to IT a few weeks later when they’ve given up trying. By that time, they’re analyzing old data.

The MySQL development team at Oracle recognized this challenge impacting customers, and MySQL HeatWave was born. The idea was simple: deploy a cloud service whereby customers of all sizes could run real-time analytics on all of their data—both transactional and historical—and enable it with “point-and-click” simplicity, and without needing the dreaded ETL.The challenge I saw when running a state IT organization is the same challenge that IT orgs have been facing ever since: data silos and the fact that MySQL is not optimized for analytics. When a business has hundreds or even thousands of MySQL instances, integrating all of that data and gleaning insights from it via analytics is painful. All too often, business users must rely on IT to perform time-consuming and costly—yet error-prone! —extract, load and transform (ETL) processes to bring all the data to one central location for analysis. Or brave business users might attempt this themselves, then reach out to IT a few weeks later when they’ve given up trying. By that time, they’re analyzing old data.

The MySQL development team at Oracle recognized this challenge impacting customers, and MySQL HeatWave was born. The idea was simple: deploy a cloud service whereby customers of all sizes could run real-time analytics on all of their data—both transactional and historical—and enable it with “point-and-click” simplicity, and without needing the dreaded ETL.

When I look at MySQL HeatWave, I consider two things—the richness of features and the ability of an average user to take advantage of these capabilities. From both perspectives, I’m impressed. As touched on above, the performance numbers are almost too good to be true. More than that, using this service is simple: no refactoring or rearchitecting of applications, no new analytics or visualization tools to learn. Just point the tool in the right direction, click a couple of times, and you have a database environment that supports online transactional processing (OLTP) and analytics.

MySQL Autopilot drives real automated operations

Some product manager once said, “You never know how your product is going to perform until it’s in the hands of paying customers.” Maybe that product manager was me in a previous part of my career. In any case, this truism is obvious to anyone who has ever launched a product.

When Oracle launched its first update to the MySQL HeatWave service in mid-2021, it focused on automating the data management lifecycle using machine learning (ML). In fact, MySQL HeatWave Autopilot was the first service I saw that automated many DBA functions that would previously consume hours a week.

Database tuning is an art form requiring both technical depth and something like clairvoyance. Deploying and provisioning databases is hard enough but tuning them is a never-ending process—one that can consume database professionals. To alleviate this, MySQL Autopilot combined deep analytics and finely tuned ML models to drive a continually optimized and always resilient MySQL database environment.

This update of MySQL HeatWave is also where I started to pay attention to Oracle’s competitive performance comparisons. Once again, the numbers initially seemed too good to be true. Suffice it to say that HeatWave outperformed every major cloud provider in analytics.

Of special note was its performance relative to the very popular Snowflake. When running TPC-H (a decision support benchmark), MySQL HeatWave showed an incredible 35x price/performance advantage over Snowflake. In terms of raw performance, HeatWave had a 6.8x advantage, and in terms of price, a 5.2x advantage. Pretty compelling, right? Benchmarks can be manipulated—or so I would think. Except that Oracle publishes its test harness on GitHub so customers can see for themselves. That shows real confidence in their capabilities.

HeatWave AutoML and multi-cloud—a natural next step

After introducing MySQL Autopilot, Oracle’s next big act with HeatWave was the integration of ML into MySQL HeatWave for model training purposes—aptly named HeatWave AutoML. This gave HeatWave the ability to automate the building, training, tuning, and explaining of ML models in real-time based on the data residing in MySQL. Again, HeatWave is democratizing machine learning by enabling this functionality for all, so that companies that don’t have teams of data scientists or TensorFlow developers can gain the same kinds of insights and automation usually reserved for larger organizations.

Additionally, Oracle embraced the notion of multi-cloud by releasing MySQL HeatWave on AWS. That means the entire MySQL HeatWave environment—core capabilities, Autopilot and ML—is available for customers to use on AWS. Oracle did this to bring the value of MySQL HeatWave to customers already using AWS who might otherwise be priced out of the service because of data egress fees. (To use HeatWave in-memory analytics, you would have to move all your data from AWS into OCI.) So, rather than force this difficult decision, Oracle stood up the HeatWave service natively—including the control plane, data plane, and console— to run on AWS. Is performance as good as it would be on OCI? No. But it’s still really good. And there is no sense that Oracle is delivering an underpowered product to ultimately convince customers to move to OCI.

If your data resides on Microsoft Azure instead, life is equally easy for you. Because Oracle and Microsoft have deployed a low-latency layer-2 network connection, you can simply use HeatWave on OCI while your applications still reside in Azure. No latency hit, no cost-prohibitive ingress and egress fees.

As an analyst who was also an Oracle customer for some time, I feel like I’m witnessing a new company with a new attitude. In the past, Oracle was not known for its emphasis on making life easy for customers. In comparison to that, its trajectory with MySQL HeatWave is a breath of fresh air.

MySQL HeatWave’s latest release: more ML goodness

In the latest update to HeatWave, the team at Oracle has doubled down on ML by driving usability and automation. The democratization of ML only really happens when ML functions are practically available to all—meaning that they don’t require a team of developers and data scientists. And Oracle has delivered even further on that promise with its latest release.

The result of Oracle’s work is automated machine learning that is highly performant and fully automated. And here’s what’s interesting: because of the highly parallelized architecture of MySQL HeatWave AutoML, these models are running on commodity CPUs. This drives down costs considerably—savings that are passed on to customers. So that price performance advantage I discussed earlier? Now it’s even better.

In addition to focusing on usability, Oracle has delivered three new ML capabilities in this latest update that are worth quickly highlighting.

1. Unsupervised anomaly detection. This feature identifies events deviating from the norm using a single algorithm. Banks looking to detect fraud, operational technology organizations looking for IoT sensor outliers, and cybersecurity teams focused on intrusion detection are all use cases that would benefit from this ability. HeatWave is the only solution to perform this detection at the local, global, and cluster levels with a single algorithm and a fully automated process. This makes it faster and more accurate than manually selecting, testing, and tuning multiple individual algorithms.
2. Recommender system availability. This capability automates the use of recommender systems—which recommend movies to watch and products to buy—in HeatWave, driving significantly faster training time coupled with a lower error rate (as compared with a public-domain solution). By comparison, other services only recommend algorithms, putting the burden on users to select the most appropriate one, and then manually tune it.
3. Multivariate time series forecasting. This feature automates the process for companies to accurately predict time series observations based on a number of variables. This would apply, for instance, if a power company is trying to determine seasonal electricity demands when considering other energy sources, weather, and so on. What once would require a team of data scientists is now just a few mouse clicks away.

Additionally, Oracle has made enhancements to MySQL Autopilot that further improve ML automation in a workload-aware way. Whether an organization is using HeatWave for OLTP, OLAP, or data lakehouse purposes, Autopilot removes mundane and tedious chores from the hands of administrators.

If the above graphic looks familiar, it is an update covering available Autopilot capabilities. The MySQL team at Oracle has added considerable functionality as MySQL Autopilot has evolved into a data management platform.

MySQL moves to a Lakehouse

Last October at CloudWorld 2022 Oracle announced MySQL HeatWave Lakehouse (currently in beta), continuing to extend HeatWave’s database capabilities. It enables customers to process and query hundreds of terabytes of data in object store in a variety of file formats, such as CSV and Parquet, as well as Aurora and Redshift export files.

Keeping with tradition, MySQL HeatWave Lakehouse delivers significantly better performance than competitive cloud database services for running queries (17X faster than Snowflake) and loading data (2.7X faster than Snowflake) on a 400TB TPC-H benchmark.

In addition, in a single query, customers can query transactional data in the MySQL database and combine it with data in the object store using standard MySQL syntax. New MySQL Autopilot capabilities that improve performance and make MySQL HeatWave Lakehouse easy to use also became available, increasing administrators’ productivity.

Is MySQL HeatWave really Oracle’s killer app?

Maybe I was being a little hyperbolic with my blog title. But MySQL HeatWave was revolutionary from the moment Oracle released it in late 2020. And it continues to separate itself from competing, but single-focus, database cloud solutions as Oracle adds functionality. MySQL HeatWave is evolving from an OLTP + in-memory analytics tool into something much bigger—if not a data management platform, then a fully featured OLTP, data analytics, machine learning, and lakehouse platform within one integrated database.

My only question is, what’s next? Considering how quickly Oracle has been innovating with HeatWave, I don’t think it will take long to find out.

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

Q&A

Q&A with Ken Muse: Designing Azure Modern Data Warehouse Solutions

• By David Ramel
• 03/28/2023

The Modern Data Warehouse (MDW) pattern makes it easier than ever to deal with the increasing volume of enterprise data by enabling massive, global-scale writes operations while making the information instantly available for reporting and insights.

As such, it’s a natural fit for cloud computing platforms and the field of DataOps, where practitioners apply DevOps principles to data pipelines built according to the architectural pattern on platforms like Microsoft Azure.

In fact, Microsoft has published DataOps for the modern data warehouse guidance and a GitHub repo featuring DataOps for the Modern Data Warehouse as part of its Azure Samples offerings.

And, speaking of GitHub, one of the leading experts in the MDW field is Ken Muse, a senior DevOps architect at the Microsoft-owned company who has published an entire series of articles on the pattern titled “Intro to the Modern Data Warehouse.” There, he goes into detail on storage, ingestion and so on.

It just so happens that Muse will be sharing his knowledge at a big, five-day VSLive! Developer Conference in Nashville in May. The title of his presentation is Designing Azure Modern Data Warehouse Solutions, a 75-minute session scheduled for May 16.

Attendees will learn:

• Define and understand how to implement the MDW architecture pattern
• How to determine appropriate Azure SQL and NoSQL solutions for a workload
• Understand how to ingest and report against high volume data

We caught up with Muse, a four-time Microsoft Azure MVP and a Microsoft Certified Trainer, to learn more about the MDW pattern in a short Q&A.

VisualStudioMagazine: What defines a Modern Data Warehouse in the Azure cloud?

Muse:
A Modern Data Warehouse combines complementary data platform services to provide a secure, scalable and highly available solution for ingesting, processing, analyzing and reporting on large volumes of data. This architectural pattern supports high-volume data ingestion as well as flexible data processing and reporting. In the Azure cloud, it often takes advantage of services such as Azure Data Lake Storage, Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Azure Cosmos DB, Azure Analysis Services, and Azure SQL.

How does the cloud improve a MDW approach as opposed to an on-premises implementation?
The cloud provides an elastic infrastructure that can dynamically scale to meet ingestion and analysis needs. Teams only pay for what they need, and they have access to virtually limitless storage and compute capacity that can be provisioned in minutes. This makes it faster and easier to turn data into actionable insights.

With an on-premises environment, the infrastructure must be sized to meet the peak needs of the application. This often results in over-provisioning and wasted resources. Hardware failures and long supply chain lead times can restrict teams from scaling quickly or exploring new approaches.

In addition, maintaining and optimizing each service can be time-consuming and complex. The cloud eliminates these issues by providing optimized environments on demand.

As developers often struggle to figure out the right tools — like SQL vs. NoSQL — for implementation, can you briefly describe what goes into making that choice, like the benefits and/or drawbacks of each?
The choice between SQL and NoSQL is often driven by the type of data you need to store and the types of queries you need to run. SQL databases are optimized for highly structured data, complex queries, strong consistency, and ACID transactions. They are natively supported in nearly every development language, making it easy to get started quickly. They can be an optimal choice for applications that commit multiple related rows in a single transaction, perform frequent point-updates, or need to dynamically query structured datasets. The strong consistency model is often easier for developers to understand. At the same time, horizontal scaling can be challenging and expensive, and performance can degrade as the database grows.

NoSQL (“not only SQL”) solutions are optimized for unstructured and semi-structured data, rapidly changing schemas, eventual consistency, high read/write volumes, and scalability. They are often a good choice for applications that need to store large amounts of data, perform frequent reads and writes, or need to dynamically query semi-structured data. They can ingest data at extremely high rates, easily scale horizontally, and work well with large datasets. They are often the best choice for graph models and understanding complex, hidden relationships.

At the same time, eventual consistency can be challenging for developers to understand. NoSQL systems frequently lack support for ACID transactions, which can make it more difficult to implement business logic. Because they not designed as a relational store, they are often not an ideal choice for self-service reporting solutions such as Power BI.

This is why the MDW pattern is important. It rely on the strengths of each tool and selects the right one for each job. It enables using both NoSQL and SQL together to support complex storage, data processing, and reporting needs.

What are a couple of common mistakes developers make in implementing the MDW pattern?
There are three common mistakes developers make in implementing the MDW pattern:

• Using the wrong storage type for ingested data: Teams frequently fail to understand the differences between Azure storage solutions such as Azure Blob Storage, Azure Files, and Azure Data Lake Storage. Picking the wrong one for the job can create unexpected performance problems.
• Forgetting that NoSQL solutions rely on data duplication: NoSQL design patterns are not the same as relational design patterns. Using NoSQL effectively often relies on having multiple copies of the data for optimal querying and security. Minimizing the number of copies can restrict performance, limit security, and increase costs.
• Using Azure Synapse Analytics for dynamic reporting: Azure Synapse Analytics is a powerful tool for data processing and analysis, but it is not designed for high-concurrency user queries. Direct querying from self-service reporting solutions such as Power BI is generally not recommended. It can provide a powerful solution for building the data models that power self-service reporting when used correctly or combined with other services.

With the massive amounts of data being housed in the cloud, what techniques are useful to ingest and report against high-volume data?
For ingesting high volume data as it arrives, queue-based and streaming approaches are often the most effective way to capture and land data. For example, Azure Event Hubs can be used to receive data, store it in Azure Data Lake Storage, and optionally deliver it as a stream to other services, including Azure Stream Analytics. For larger datasets, it can be advisable to store the data directly into Azure Blob Storage or Azure Data Lake Storage. The data can then be processed using Azure Data Factory, Azure Synapse Analytics, or Azure Databricks. They key is to land the data as quickly as possible to minimize the risk of data loss and enable downstream rapid analysis.

For reporting, it’s important to optimize the data models for the queries that will be run. The optimal structures for reporting are rarely the same as those used for ingestion or CRUD operations. For example, it’s often more efficient to denormalize data for reporting than it is to store it in a normalized form. In addition, column stores generally perform substantially better than row-based storage for reporting. As a result, separating the data capture and data reporting aspects can help optimize the performance of each.

How can developers support high volumes of read/write operations without compromising the performance of an application?
An important consideration for developers is the appropriate separation of the read and write operations. When read and write operations overlap, it creates contention and bottlenecks. By separating the data capture and data reporting aspects, you can optimize the performance of each. You can also select services which are optimized for that scenario, minimizing the development effort required.

For applications that need to support CRUD (create, read, update, delete) operations, this can require changing the approach. For example, it may be necessary to use a NoSQL solution that supports eventual consistency. It may also be necessary to persist the data in multiple locations or use change feeds to propagate updates to other services.

In other cases, tools such as Azure Data Factory may be more appropriate. It can periodically copy the data to a different data store during off-peak hours. This can help minimize the impact of the changes to the application. This can be important when the needs of the application change suddenly or when the application does not have to provide up-to-the-moment reporting data.

What are some key Azure services that help with the MDW pattern?
The key services used in the MDW pattern are typically Azure Data Lake Storage Gen2, Azure Synapse Analytics, Azure Databricks, Azure SQL, and Azure Event Hubs.

That said, there are many other services that can be used to support specific application and business requirements within this model. For example, Azure Machine Learning can be used to quickly build insights and models from the data. Azure Cosmos DB can be used to support point-queries and updates with low latency. Services like Azure Purview can be used to understand your data estate and apply governance. The MDW pattern is about understanding the tradeoffs between the different services to appropriate select ones that support the business requirements.

As AI is all the rage these days, do any of those Azure services use hot new technology like large language models or generative AI?
Absolutely! A key part of the Modern Data Warehouse pattern is supporting machine learning, and that includes generative AI and new techniques that development teams might be working to create themselves.

Azure’s newest offering, Azure OpenAI Service, is a fully managed service that provides access to the latest state-of-the-art language models from OpenAI. It is designed to help developers and data scientists quickly and easily build intelligent applications that can understand, generate, and respond to human language.

In addition, Azure recently announced the preview of the ND H100 v5 virtual machine series. These are optimized to support the training of large language models and generative AI. These virtual machines boost the performance for large-scale deployments by providing eight H100 Tensor Core CPUs, 4th generation Intel Xeon Processors, and high-speed interconnects with 3.6 TBps of bidirectional bandwidth among the eight local GPUs. You can learn more here.

Article originally posted on InfoQ. Visit InfoQ

At the recent QCon London conference, Mathew Lodge, CEO of DiffBlue, gave a presentation on the advancements in artificial intelligence (AI) for writing code. Lodge highlighted the differences between Large Language Models and Reinforcement Learning approaches, emphasizing what both approaches can and can’t do. The session gave an overview of the state of the current state of AI-powered code generation and its future trajectory. 

In his presentation, Lodge delved into the differences between AI-powered code generation tools and unit test writing tools. Code generation tools like GitHub Copilot, TabNine, and ChatGPT primarily focus on completing code snippets or suggesting code based on the context provided. These tools can greatly speed up the development process by reducing the time and effort needed for repetitive tasks. On the other hand, unit test writing tools such as DiffBlue aim to improve the quality and reliability of software by automatically generating test cases for a given piece of code. Both types of tools leverage AI to enhance productivity and code quality but target different aspects of the software development lifecycle.

Lodge explained how code completion tools, particularly those based on transformer models, predict the next word or token in a sequence by analyzing the given text. These transformer models have evolved significantly over time, with GPT-2, one of the first open-source models, being released in February 2019. Since then, the number of parameters in these models has scaled dramatically, from 1.5 billion in GPT-2 to 175 billion in GPT-3.5, released in November 2022.

OpenAI Codex, a model with approximately 5 billion parameters used in GitHub CoPilot, was specifically trained on open-source code, allowing it to excel in tasks such as generating boilerplate code from simple comments and calling APIs based on examples it has seen in the past. The one-shot prediction accuracy of these models has reached levels comparable to explicitly trained language models. Unfortunately, information regarding the development of GPT-4 remains undisclosed. Both training data and information around the number of parameters is not published which makes it a black box. 

Lodge also discussed the shortcomings of AI-powered code generation tools, highlighting that these models can be unpredictable and heavily reliant on prompts. As they are essentially statistical models of textual patterns, they may generate code that appears reasonable but is fundamentally flawed. Models can also lose context, or generate incorrect code that deviates from the existing code base calling functions or APIs which do not exist. Lodge showed an example of code for a so-called perceptron model which had two difficult to spot bugs in them which essentially made the code unusable. 

GPT-3.5, for instance, incorporates human reinforcement learning in the loop, where answers are ranked by humans to yield improved results. However, the challenge remains in identifying the subtle mistakes produced by these models, which can lead to unintended consequences, such as the ChatGPT incident involving the German coding company OpenCage.

Additionally, Large Language Models (LLMs) do not possess reasoning capabilities and can only predict the next text based on their training data. Consequently, the models’ limitations persist regardless of their size, as they will never generate text that has not been encoded during their training. Lodge highlighted that these problems do not go away, no matter how much training data and parameters are actually used during the training of these models. 

Lodge then shifted the focus to reinforcement learning and its application in tools like DiffBlue. Reinforcement learning differs from the traditional approach of LLMs by focusing on learning by doing, rather than relying on pre-existing knowledge. In the case of DiffBlue Cover, a feedback loop is employed where the system predicts a test, runs the test, and then evaluates its effectiveness based on coverage, other metrics, and existing Java code. This process allows the system to iteratively improve and generate tests with higher coverage and better readability, ultimately resulting in a more effective and efficient testing process for developers. Lodge also mentioned that their representation of test coverage allows them to only run relevant tests when changing code, resulting in a decrease of about 50% of testing costs.

To demonstrate the capabilities of DiffBlue Cover, Lodge conducted a live demo featuring a simple Java application designed to find owners. The application had four cases for which tests needed to be created. Running entirely on a local laptop, DiffBlue Cover generated tests within 1.5 minutes. The resulting tests appeared in IntelliJ as a new file, which included mocked tests for scenarios such as single owner return, double owner return, no owner, and an empty array list.

In conclusion, the advancements in AI-powered code generation and reinforcement learning-based testing, as demonstrated by tools like DiffBlue Cover, have the potential to greatly impact the software development and testing landscape. By understanding the strengths and limitations of these approaches, developers and architects can make informed decisions on how to best utilize these technologies to enhance code quality, productivity, and efficiency while reducing the risk of subtle errors and unintended consequences.
 

Article originally posted on InfoQ. Visit InfoQ

The PyTorch Foundation recently released PyTorch version 2.0, a 100% backward compatible update. The main API contribution of the release is a compile function for deep learning models, which speeds up training. Internal benchmarks on 163 open-source AI projects showed that the models ran on average 43% faster during training.

Plans for the 2.0 release were announced at the PyTorch Conference in December 2022. Besides the new compile function, the release also includes performance improvement for Transformer-based models, such as large language models and diffusion models, via a new implementation of scaled dot product attention (SDPA). Training on Apple silicon is accelerated via improved Metal Performance Shaders (MPS), now with 300 operations implemented in MPS. Besides the core release, the domain libraries, including TorchAudio, TorchVision, and TorchText, were updated with new beta features. Overall, the 2.0 release includes over 4,500 commits from 428 developers since the 1.13.1 release. According to the PyTorch Foundation blog,

We are excited to announce the release of PyTorch® 2.0 which we highlighted during the PyTorch Conference on 12/2/22! PyTorch 2.0 offers the same eager-mode development and user experience, while fundamentally changing and supercharging how PyTorch operates at compiler level under the hood with faster performance and support for Dynamic Shapes and Distributed.

In his keynote speech at the PyTorch Conference 2022, PyTorch co-creator Soumith Chintala pointed out that thanks to increases in GPU compute capacity, many existing PyTorch workloads are constrained by memory bandwidth or by PyTorch framework overhead. Previously the PyTorch team had addressed performance problems by writing some of their core components in C++; Chintala described PyTorch as “basically a C++ codebase,” and said that he “hates” contributing to the C++ components.

The new compile feature is based on four underlying components written in Python:

• TorchDynamo – performs graph acquisition by rewriting Python code representing deep learning models into blocks of computational graphs
• AOTAutograd – performs “ahead of time” automatic differentiation for the backprop step
• PrimTorch – canonicalizes the over 2k PyTorch operators down to a fixed set of around 250 primitive operators
• TorchInductor – generates fast hardware-specific backend code for accelerators

To demonstrate the performance improvements and ease of use of the compile function, the PyTorch team identified 163 open-source deep learning projects to benchmark. These included implementations of a wide variety of tasks including computer vision, natural language processing, and reinforcement learning. The team made no changes to the code besides the one-line call to the compile function. This single change worked in 93% of the projects, and the compiled models ran 43% faster when trained on NVIDIA A100 GPUs.

In a Hacker News discussion about the release, one user noted:

A big lesson I learned from PyTorch vs other frameworks is that productivity trumps incremental performance improvement. Both Caffe and MXNet marketed themselves for being fast, yet apparently being faster here and here by some percentage simply didn’t matter that much. On the other hand, once we make a system work and make it popular, the community will close the performance gap sooner than competitors expect. Another lesson is probably old but worth repeating: investment and professional polishing [matter] to open source projects.

The PyTorch code and version 2.0 release notes are available on GitHub.