What is data analysis and why is it important?

Data analysis is the process of evaluating data using analytical and statistical tools to discover useful information and help you make business decisions. There are several methods for analyzing data, including data mining, text analysis, business intelligence, and data visualization.Not only does the most complex application data needs analysis but even a simple email database needs analysis and cleansing.

How is data analysis done?
Data analysis is part of the broader process of getting business intelligence. The process includes one or more of the following steps:

Defining Goals: Any research should start with a set of well-defined business goals. Most of the decisions made in the rest of the process depend on how clearly the research objectives are formulated.

Asking questions: An attempt was made to ask a question in a problem area. For example, do red sports cars crash more often than others?

Gathering information: Data relevant to this issue should be obtained from appropriate sources. In the example above, data can be obtained from a variety of sources, including: DMV or police incidents, insurance claims, and hospitalization records. When data is collected through surveys, a questionnaire for subjects must be submitted. Questions should be modeled appropriately for the statistical method used.

Data Processing: Raw data can be collected in several different formats. The collected data must be cleaned and transformed so that the data analysis tools can import it. In our example, we can retrieve DMV crash reports as text files, claims from a relational database, and hospital admissions as APIs. The data analyst must combine these different forms of data and transform them into a form suitable for analysis tools.

Data Analysis: In this step, the cleaned and aggregated data is imported into the analysis tools. These tools allow you to explore your data, find patterns in it, and ask and answer what-if questions. It is the process by which the data gathered in research is made meaningful through the correct application of statistical methods.

Overall, data analysis provides an understanding that businesses need to make effective and efficient decisions. Combined with Data Analytics, they have a good understanding of the needs and capabilities of the company. Data analysis applications are very broad. Big data analytics can optimize the performance of verticals across industries.

Increased productivity enables companies to thrive in an ever-evolving competitive environment and to be successful in many areas. Environmental protection and crime prevention are some of the applications of data analysis. Data analysis apps now look endless.

Every day more and more data is collected, which opens up new opportunities for applying data and improving society and the world.

Reference: www.mastersindatascience.org/learning/what-is-data-analytics/

Source Prolead brokers usa

Explainable AI for Medical Images

Most of what goes by the name of Artificial Intelligence (AI) today is actually based on training and deploying Deep Learning (DL) models. Despite their impressive achievements in fields as diverse as image classification, language translation, complex games (such as Go and chess), speech recognition, and self-driving vehicles, DL models are inherently opaque and unable to explain their predictions, decisions, and actions.

This is not a critical issue for several applications (such as movie recommendation systems or news/social media feed customization, for example) where the end user will evaluate the quality of the AI based on the results it produces, make occasional adjustments to help it improve future results (e.g., by rating additional movies), or move away from that product/app. There is rarely a need to require an explanation for the AI’s decisions when there is very little at stake.

However, for high-stakes situations and mission-critical applications – such as self-driving vehicles, criminal justice decisions, financial systems, and healthcare applications – explainability might be considered crucial. It is not enough to have an AI provide a solution, decision, or diagnosis; it is often necessary (and in some cases, absolutely essential) to explain the process behind such decisions. The emerging field of XAI (eXplainable Artificial Intelligence) addresses this need and offers several methods to provide some level of explanation to deep learning AI solutions.

In this blog post, we focus primarily on the human factors of XAI: What needs to be explained and how can this be communicated to the (human) user of an AI solution?

Fig 1 – Human factors of XAI: an explainable model requires a suitable interface

Recognizing that explainability is desirable when deploying AI solutions in some areas of human activity is only the beginning. In order to understand the why and how behind an AI model’s decisions and get a better insight into its successes and failures, an explainable model must explain itself to a human user through some type of explanation interface (Figure 1). Ideally such interface should be rich, interactive, intuitive, and appropriate for the user and task.

In the field of image classification, a common interface for showing the results of XAI techniques consists of overlaying the “explanation” (usually in the form of a heat map or saliency map) on top of the image. This can be helpful in determining which areas of the image the model deemed to be most relevant for its decision-making process. It can also assist in diagnosing potential blunders that the DL model might be making, producing results that are seemingly correct but reveal that the model was “looking at the wrong places.” Classical examples include the husky vs. wolf image classification algorithm that was, in fact, a “snow detector”, and an image classification solution where the tench, a large fish species, is often identified by human fingers.

In radiology, there is a well-known case where models used to detect pneumonia in chest X-rays had learned to detect a metal token that radiology technicians place on the patient in the corner of the image field of view at the time they capture the image, which in turn identified the source hospital, causing the models to perform well in images from the hospital they were trained on, and poorly in images from other hospitals with different markers.

XAI is often presented as an “all-or-nothing” addition to regular AI, leading to potentially false dichotomies – such as the trade-offs between accuracy and explainability (which suggests that in order to get more of one you must sacrifice the other, which is not universally true) – or vague questions based on abstract scenarios, such as “Would you pay more for AI that explains itself (and performs at the same level as the baseline solution)?”

Fig 2 – XAI as a gradual approach: in addition to the model’s prediction, different types of supporting information can be added to explain the decision.

We choose to see explainability as a gradual process instead (Figure 2), where an AI system that predicts a medical condition from a patient’s chest x-ray might use gradually increasing degrees of explainability: (1) no explainability information, just the outcome/prediction; (2) adding output probabilities for most likely predictions, giving a measure of confidence associated with them; (3) adding visual saliency information describing areas of the image driving the prediction; (4) combining predictions with results from a medical case retrieval (MCR) system and indicating matched real cases that could have influenced the prediction; and (5) adding computer-generated semantic explanation [5].

There is no universally accepted taxonomy of XAI techniques. In this blog post we use the categorization of the field adopted by the authors of a recent survey paper on the topic, which breaks down the field into scope, methodology and usage, as follows:

• Scope: Where is the XAI method focusing on?
  • The scope could be local (where the focus is on individual data instances) or global (where the emphasis is on trying to understand the model as a whole)
• Methodology: What is the algorithmic approach?
  • The algorithm could focus on: (a) gradients back-propagated from the output prediction layer to the input layer; or (b) random or carefully chosen changes to features in the input data instance, also known as perturbations.
• Usage: How is the XAI method developed?
  • Explainability might be intrinsic to the neural network architecture itself (which is often referred to as interpretability) or post-hoc, where the algorithm is applied to already trained networks.

Some post-hoc XAI algorithms have become popular in recent years, among them Grad-CAM (Gradient-weighted Class Activation Mapping) and LIME (Local Interpretable Model-agnostic Explanation). Grad-CAM is gradient-based, has local scope, and can be used for computer vision tasks. LIME is perturbation-based, has both local and global scope, and can be used with images, text, or tabular data.

Here is an example of how to use MATLAB to produce post-hoc explanations (using Grad-CAM and image LIME) for a medical image classification task, defined as follows:

Given a chest x-ray (CXR), our solution should classify it into Posteroanterior (PA) or Lateral (L) view.  

The dataset (PadChest) can be downloaded from https://bimcv.cipf.es/bimcv-projects/padchest/ .

Both methods (gradCAM and imageLIME) are available as part of the MATLAB Deep Learning toolbox and require a single line of code to be applied to results of predictions made by a deep neural network (plus a few lines of code to display the results as a colormap overlaid on the actual images).

Figures 3 and 4 show representative results. The Grad-CAM results (Fig 3) suggest that the network is using information from the borders and corners of the image to make a classification decision. The LIME results (Fig 4) tell a rather different story. In both cases, we are left wondering: to what extent did either method help explain the model’s decision?

Fig 3 – Example of result: Grad-CAM for CXR classification task.

Fig 4 – Example of result: image LIME for CXR classification task.

Here is another example where we applied Grad-CAM and image LIME to results of a binary skin lesion classifier.

Figures 5 and 6 show representative results. They are organized in a confusion matrix-like fashion (with one case each of true positive (TP), true negative (TN), false positive (FP), and false negative (FN)). They reveal fascinating aspects (and limitations) of these post-hoc techniques:

• For easy images (such as the true positive on the bottom-right corner), both techniques perform extremely well and suggest that the model was “looking at the right place within the image”
• For the false positive on the bottom-left corner, both techniques show that the model “learned” the ruler on the left-hand-side of the image (in addition to taking into account aspects from the lesion region as well), which might have explained the incorrect prediction
• For the images on the top row, both techniques suggest that the insights are not as clear (at least to a layperson) as the images on the bottom row; one of the problematic aspects of these post-hoc techniques is actually exemplified here, since they are equally comfortable explaining correct as well as incorrect decisions in a similar way.

Fig 5 – Example of result: Grad-CAM for skin lesion classification task.

Fig 6 – Example of result: image LIME for skin lesion classification task.

Deep Learning models are opaque and, consequently, have been deemed “inadequate” for several applications (since usually they can make predictions without explaining how they arrived at the result, or which factors had greater impact). Explainable AI (XAI) techniques are possible ways to inspire trust in an AI solution.

In this blog post we have used MATLAB to show how post-hoc explanation techniques can be used to show which parts of an image were deemed most important for medical image classification tasks. These techniques might be useful beyond the explanation of correct decisions, since they also help us identify blunders, i.e., cases where the model learned the wrong aspects of the images. 

XAI is an active research area, and new techniques and paradigms are likely to emerge in the near future. If you’re interested in learning more about XAI and related issues, see the  recommended resources below.

Source Prolead brokers usa

It’s Insights Monetization, Not Data Monetization

Okay, this is on me.  I own it.  My blog “Why Data Monetization is a Waste of Time” created quite a stir.  I loved it.  I’d like to take this “data monetization” conversation one step further:

Most organizations should not be focused on data monetization; instead, they should be focused on insights monetization

When I published my first book “Big Data” in September, 2013, I referred to the fourth stage of the “Big Data Business Model Maturity Index as the “Data Monetization” phase.  However, it wasn’t until my latest book “The Economics of Data, Analytics, and Digital Transformation” that I published the end of last year, that I finally got the concept right.  For most organizations, it shouldn’t be “Data Monetization.”  Instead, “Insights Monetization” – customer, product, and operational insights or predicted propensities – is more appropriate (Figure 1).

Figure 1: Data & Analytics Business Maturity Index

The “Insights Monetization” phase of the Data & Analytics Business Maturity Index is described as such:

The Insights Monetization phase requires business leadership to envision (using design thinking) how the organization can leverage and blend their wealth of customer, product, and operational insights (predicted propensities) to create new monetization opportunities including new markets and audiences, new products and services, new channels and partnerships, new consumption models, etc.

That doesn’t mean to preclude data monetization – or the direct selling of one’s data – as an option, but data monetization has a very different context than insights monetization.  Let me explain.

The problem with the term “Data Monetization” is that for many organizations, this implies the direct selling of the organization’s data.  And there are certainly companies out there that do sell data.  Nielsen, Acxiom, Experian, Equifax and CoreLogic are companies whose business model is the acquisition, aggregation, and selling of third-party data.  For example, Figure 2 shows the personal data that one can buy from Acxiom.  Yea, sort of scary.

Figure 2:  Source: “Here are the data brokers quietly buying and selling your personal …”

Selling data requires a significant organization to acquire, cleanse, align, package, market, sell, support, and manage the data for external consumption.  And there is a myriad of growing legal and privacy concerns to navigate, so a pretty decent legal team will be required as well.

For other organizations, data monetization amounts to creating data services that facilitate the exchange of an organization’s data in exchange for something of value from another organization. Walmart’s Retail Link® is an example of this sort of “data monetization.”

Walmart’s Retail Link® gives Walmart suppliers – think Consumer Packaged Goods (CPG) companies like Procter & Gamble, PepsiCo, and Unilever – access to Walmart’s “point of sale” (POS) data. Retail Link provides suppliers access to the supplier’s product sell-through (sales) data by SKU, by hour, by store. Suppliers can also get on-hand inventory by SKU, as well as gross margin achieved, inventory turns, in-stock percentages, and Gross Margin Return on Inventory Investment (Figure 3).

Figure 3: Sample report courtesy of  Trend Results[1]

This is a great example of Data Monetization. Unfortunately, not all organizations have the clout and financial and technology resources of a Walmart to dictate this sort of relationship.  Walmart invests a significant amount of time, money and people resources to on-board, support, maintain, and upgrade Retail Link.  In that aspect, Walmart looks and behaves like an enterprise software vendor.

But for organizations that lack the clout, finances, and technology expertise of a Walmart, there are other “monetization” options.

Insights Monetization is about leveraging the customer, product, and operational insights (predicted propensities) buried in your data sources to optimize and/or reengineer key business and operational processes, mitigate (compliance, regulatory, and business) risks, create new revenue opportunities (such new products, services, audiences, channels, markets, partnerships, consumption models, etc.), and construct a more compelling, differentiated customer experience.

To drive “Insights Monetization” requires some key concepts and techniques.

(1) Nanoeconomics.  Nanoeconomics is the economics of individualized human and/or device predicted propensities.  Nanoeconomics helps organizations transition from overly generalized decisions based upon averages to precision decisions based upon the predicted propensities, patterns, and trends of individual humans or devices (Figure 4).

Figure 4:  Nanoeconomics to Transform Organizational Decision Making

Remember, making decisions based on averages at best yield average results.  We can do better than average by leveraging nanoeconomics to make precision policy and operational decisions.

(2) Analytic Profiles provide a model for capturing and codifying the organization’s customer, product, and operational analytic insights (predicted propensities) in a way that facilities the sharing and refinement of those analytic insights across multiple use cases (Figure 5).

Figure 5: Analytic Profiles

An Analytic Profile capture of metrics, predictive indicators, segments, scores, and business rules that codify the behaviors, preferences, propensities, inclinations, tendencies, interests, associations and affiliations for the organization’s key business entities such as customers, patients, students, athletes, jet engines, cars, locomotives, CAT scanners, and wind turbines.

(3) Use Cases are clusters of Decisions around a common Key Performance Indicator (KPI) where Decisions are a conclusion or resolution reached after analysis that leads to an informed action.  Sample use cases include reduce customer attrition, improve operational uptime, and optimize asset utilization. And the application of the Analytic Profiles can be used to help optimize or address the organization’s top priority use cases (Figure 6).

Figure 6: Role of Analytic Profiles in Optimizing Use Cases

The power of use cases are:

• Readily identifiable by the business stakeholders (because they live them every day)
• Resolution or optimization is a source of quantifiable business and operational value (improving, reducing, or optimizing the use cases can have material business and operational value)
• Encourages tight collaboration between business stakeholders and the data & analytics teams to properly define the objectives, benefits, value statements, and potential impediments of the use case, identify the metrics and KPIs against which use case progress and success will be measured, and codify the associated costs of use case False Positives and False Negatives (see “The Art of Thinking Like a Data Scientist” methodology).
• Provides the basis for a use case-by-use case development and implementation approach – driven by Return on Investment (ROI) – to building the organization’s data and analytic assets (see the “Schmarzo Economic Digital Asset Valuation Theorem“).

The good news:  organizations have a bounty of use cases.  The bad news: organizations don’t fail due to a lack of use cases; they fail because they have too many. 

Organizations tend to fail because they peanut butter their precious data and analytics resources across too many poorly defined use cases, and don’t get the prerequisite business stakeholder buy-in to actually deploy or use the resulting analytics.  Focus baby, focus!

Most organizations should not be focused on data monetization; instead, they should be focused on insights monetization

While there will be examples of companies who can successfully “sell their data” (Nielsen, Walmart), for most organizations their monetization journey will be more about applying their customer, product, and operational insights (or predicted propensities) to optimize their top priority business and operational use cases.

Insights Monetization is about leveraging the customer, product, and operational insights (predicted propensities) buried in your data sources to optimize and/or reengineer key business and operational processes, mitigate (compliance, regulatory, and business) risk create new revenue opportunities (such new products, services, audiences, channels, markets, consumption models, etc.) and create a more compelling, differentiated customer experience.

Heck, it only took me about 7 years to get the distinction between data monetization and insights monetization right.  As Homer would say, DOH!

[1] Trend Results is in no way associated with or endorsed by Wal-Mart Stores, Inc. All references to Wal-Mart Stores, Inc. trademarks and brands are used in strict accordance with the Fair Use Doctrine and are not intended to imply any affiliation of Trend Results with Wal-Mart Stores, Inc. Retail Link is a registered trademark of Wal-Mart Stores, Inc.

Source Prolead brokers usa

Intro to the E-R Diagram

• E-R diagrams capture meaning in your data.
• This high-level visual tool identifies core components of an enterprise
• Simple steps to follow to create the diagram.

Entity-Relationship (E-R) Modeling is one approach to visualize what story your data is trying to tell. This goal of this predecessor to object modeling (e.g. UML or CRC cards) is to give you a high-level, graphical view of the core components of an enterprise—the E-R diagram. An E-R diagram (sometimes called a Chen diagram, after its creator, Peter Chen) is a conceptual graph that captures meaning rather than implementation [1]. Once you have the diagram, you can convert it to a set of tables.

Entity Relationship Diagrams can quickly become very complex and can seem overwhelming to look at for the first time. However, the diagram is built block by block, based on elements which you define. If you’re familiar with your data, and you know some basic E-R diagram symbols (more on that below), you can build an E-R diagram following a few simple rules.

The starting point for making an E-R diagram is to identify a few key items from users. Some key questions to ask [2]:

• What data needs to be kept?
• What queries do we need to ask?
• What business rules should we build in? For example, if the BANK ACCOUNT table has one column for owner, then you have committed to having one owner per bank account.

Next, you want to identify entities. An entity is something about which data is collected, stored or maintained [3]. Look for concrete objects in the problem domain that you can describe with a noun. For example, your core components for a telecommunications company might be WORKERS, CUSTOMERS, CELL PHONES and SERVICE PLANS. You’re looking to describe objects that are clearly distinguishable from other objects, like people, places, and things. “Things” can also be an abstraction like project names or departments.

List the attributes of each entity.  An attribute is a characteristic of an entity—usually a noun. Attributes describe parts of an entity and should be indivisible [ideally] single-valued. For example, TEACHER might have the attributes “name”, “title” and “specialty”. These tend to (more or less) become the fields. Although these should ideally be single (like social security number), they can be layered/composite (like city/state/zip). The domain tells us what values are permissible for an attribute. For example, (John, 123 Front Street, 9999).

Putting the above two components together, each ENTITY is described by a set of attributes. For example:

TEACHER (entity): Id, Name, Address, Parking Decal Number (attributes).

Build Relationships: A relationship is an association between two or more entities. What relationships are present in your data? Employee-supervisor? Teacher-student? Product-Consumer? Sometimes you may have to make a choice if entities and relationships aren’t clear cut. For example, “customer orders” could be an entity, or it could be a relationship.

From this basic information, you can create a simple E-R diagram.  First, you need to be familiar with the meaning of a few basic shapes. Like flow charts, each shape in the E-R diagram has a specific meaning:

• Rectangle: Entity sets.
• Double rectangle: weak entity: one that exists only as the result of another entity. Examples dependents, invoices, transactions.    
• Diamond: Relationship.
• Oval: Attribute.
• Double Oval: attributes that can have multiple values.
• Dashed oval: An attribute that can be derived from others. In other words, it can be calculated from other “stored” attributes in the database. For example, “age” can be derived from “date of birth” and the current date.

The above diagram shows the basic shapes. Start with a few core entities, and build out. As you should follow a few placement rules, like keeping attributes above entities, you’ll want to use software (even something simple like PowerPoint will work) to create one. A few tips for making the E-R diagram:

• Common practice is for entities to be described by a single noun.
• Attributes should be kept above the entity if possible.
• Connect relationship diamonds from the left and right points.
• Use straight connecting lines: do not use arrows.
• Underline key attributes–unique identifiers like social security number or driver license number.

References

Image of Basic E-R diagram: By Author

[1]  Database Design   http://jcsites.juniata.edu/faculty/rhodes/dbms/ermodel.htm

[2] Entity-Relationship modeling http://pld.cs.luc.edu/database/ER.html

[3] Drawing the Entity-Relationship Diagram https://www.csee.umbc.edu/portal/help/oracle8/java.815/a64686/05_de…

Source Prolead brokers usa

AI powered cyberattacks – threats and defence

Source Prolead brokers usa

A Few Useful Techniques for Business Forecasting

Source Prolead brokers usa

Technology Used for Backend Development in 2021

Have you ever visited a website that is simply beautiful? No other words to describe it. It has the perfect aesthetic, and everything seems in the right place. And then you start browsing it. Like the world’s most naive Tinder user you discover it’s all a façade. Nothing seems to work, and the website itself takes too much time to load. Naturally, you leave the page and take your browsing time elsewhere. In real life, catfish experiences at least give you a story to tell. On the internet, all it gives you is a bounce. 

This simple example highlights why backend development is a crucial element of all websites. Users may only see the superficial aspect of a website – the frontend – but what matters is the behind-the-scenes work. If a website doesn’t work correctly, they’ll leave. And as a result, companies end up losing a potential customer.  

Technology keeps improving, and websites, too. There are more tools and options available for companies to develop great websites. And backend developers, who are responsible for the server-side of the platform, have a rich amount of tools that help them improve and offer the user the best experience possible. Let’s find out more about the technology used for backend development in 2021. 

Table of contents: 

1. What is backend development
2. Best backend development tools in 2021
3. To sum it up

What is backend development? 

One thing is how a website looks. Another entirely different thing is how a website works. Both aspects go hand in hand in ensuring the success of a platform. In general terms, frontend development focuses on the client-side of a web application; on what the user sees. Backend focuses on the server-side of a web application; how it works.

Backend engineering specializes in the server-side and everything that communicates between the database and the browser. It ensures that a web application has good performance and speed. Additionally, backend developers are constantly monitoring and organizing database information to make it secure. 

As the backend implements all users’ requests via the frontend part, it is fundamental that developers choose the right backend technology that helps them build a solid web application. 

But what tools should backend developers use in 2021? 

Top 8 backend development tools in 2021

Best backend programming languages

1. PHP

Hypertext Preprocessor, or PHP as popularly known, is a server-side scripting language and one of the most used programming languages for backend development. It is used to manage dynamic content, databases, tracking sessions, and build different websites. Additionally, its cross-platform compatibility, easy integration with HTML, CSS, JavaScript, community support, and security make it one of the most preferred programming languages among developers.

What is PHP used for:  

• Collect form data
• Generate dynamic page content
• Send and receive cookies
• Write command-line and server-side scripting
• Write desktop applications

2. Java

Java is an object-oriented programming language designed to have as few implementation dependencies as possible. It was created 26 years ago, and it continues to be one of the most popular programming languages. Its good reputation relies on the fact that it’s multipurpose; it can be used for desktop, web, and android development.

What is Java used for:

• Image processing
• GUI based programs
• Networking
• Desktop and mobile computing
• Develop artificial intelligence

Best backend frameworks

3. Laravel

Laravel is an open-source web framework that follows the Model-View-Controller (MVC) architectural pattern, making it easier for developers to start projects as they have their code better structured. This PHP framework also provides tools such as Artisan, pre-installed Object-Oriented libraries, Object Relational Mapping, among others. It is mainly used to build custom web apps and automatizes specific processes such as routing, templating HTML, and authentication.

What is Laravel used for:

• Task scheduling
• Automation testing
• Authorization
• Error handling
• URL routing configuration 

4. Django

Django is a Python-based open-source framework that allows developers to build websites efficiently. When a backend is created with Python, Django is used as an additional tool. It provides developers with great features such as extensibility, rapid development, security, scalability, among others. Businesses, in particular, are using Django to expand web development areas such as social networking platforms and content management systems.

What is Django used for?

• Customizable applications
• Secure foundation
• Scalable applications
• Builds SaaS applications
• Creates multiple user roles platforms

Best backend tech stacks

5. MEAN

MEAN stands for MongoDB, Express.js, AngularJS, and Node.js, and it’s a free and open-source JavaScript software stack suitable for building dynamic websites and applications. When developers use the MEAN stack, the backend can be created faster. This stack is a great backend alternative as it allows developers to create a complete website using JavaScript only. From the client to the server and from the server to the database, everything is based on JS.

What is MEAN used for?

• Web applications/sites
• Workflow management tools
• News aggregation sites
• Calendar apps
• Interactive forums

6. MERN

The MERN stack includes MongoDB as a database, Express.js as a backend framework, React as a library for front end, and Node.js. So, the only difference from the MEAN stack is the replacement of Angular with React. The MERN architecture allows developers to easily build a 3-tier architecture (frontend, backend, database) entirely using JS and JSON. This tech stack allows data to flow naturally from the front and back, making it easy to build on. 

What is MERN used for? 

• Cloud-native interfaces
• Social products
• Dynamic websites
• News aggregation
• Calendar applications 

Best backend web servers

7. Apache

Apache is an open-source web server that is currently used by at least 55,698,064 websites. It offers multiple features such as Loadable Dynamic Modules, Multiple Request Processing modes, CGI support, User and Session Tracking, among many others. Additionally, Apache is compatible with almost all operating systems like Linux, macOS, Windows, etc.

What is Apache used for?

• Session tracking
• Geolocation based on IP address
• Handling of static files
• Supports HTTP/2
• Auto-indexing 

8. NGINX

NGINX is an open-source web server used for reverse proxying, load balancing, mail proxying, and more. It’s built to offer low memory usage and high concurrency. Instead of creating new processes for every web request, NGINX uses an asynchronous event-driven approach to handle requests in a single thread. With this web server, one developer can control multiple worker processes. Additionally, each request can be executed concurrently without blocking other requests as NGINX is asynchronous. 

What is NGINX used for?

• Websockets
• Handling of static files, index files, and auto-indexing
• Media streaming
• Proxy server for email
• Microservices support 

To sum it up

Backend engineering is decisive to make a web application successful. These 8 backend development tools provide everything a developer needs to work on the functional and logical aspects of websites and applications. From the core languages to frameworks and web servers, it is fundamental that development teams understand the company’s priorities and goals to decide which tools will help them create a successful system architecture and consequently provide valuable solutions. 

Source Prolead brokers usa

How to Fine-Tune BERT Transformer with spaCy 3

Since the seminal paper “Attention is all you need” of Vaswani et al, Transformer models have become by far the state of the art in NLP technology. With applications ranging from NER, Text Classification, Question Answering or text generation, the applications of this amazing technology are limitless.

More specifically, BERT — which stands for Bidirectional Encoder Representations from Transformers— leverages the transformer architecture in a novel way. For example, BERT analyses both sides of the sentence with a randomly masked word to make a prediction. In addition to predicting the masked token, BERT predicts the sequence of the sentences by adding a classification token [CLS] at the beginning of the first sentence and tries to predict if the second sentence follows the first one by adding a separation token[SEP] between the two sentences.

BERT Architecture

In this tutorial, I will show you how to fine-tune a BERT model to predict entities such as skills, diploma, diploma major and experience in software job descriptions. If you are interested to go a step further and extract relations between entities, please read our article on how to perform joint entities and relation extraction using transformers.

Fine tuning transformers requires a powerful GPU with parallel processing. For this we use Google Colab since it provides freely available servers with GPUs.

For this tutorial, we will use the newly released spaCy 3 library to fine tune our transformer. Below is a step-by-step guide on how to fine-tune the BERT model on spaCy 3. The code along with the necessary files are available in the Github repo.

To fine-tune BERT using spaCy 3, we need to provide training and dev data in the spaCy 3 JSON format (see here) which will be then converted to a .spacy binary file. We will provide the data in IOB format contained in a TSV file then convert to spaCy JSON format.

I have only labeled 120 job descriptions with entities such as skills, diploma, diploma major, and experience for the training dataset and about 70 job descriptions for the dev dataset.

In this tutorial, I used the UBIAI annotation tool because it comes with extensive features such as:

• ML auto-annotation
• Dictionary, regex, and rule-based auto-annotation
• Team collaboration to share annotation tasks
• Direct annotation export to IOB format

Using the regular expression feature in UBIAI, I have pre-annotated all the experience mentions that follows the pattern “\d.*\+.*” such as “5 + years of experience in C++”. I then uploaded a csv dictionary containing all the software languages and assigned the entity skills. The pre-annotation saves a lot of time and will help you minimize manual annotation.

UBIAI Annotation Interface

For more information about UBIAI annotation tool, please visit the documentation page and my previous post “Introducing UBIAI: Easy-to-Use Text Annotation for NLP Applications”.

The exported annotation will look like this:

MS B-DIPLOMA
in O
electrical B-DIPLOMA_MAJOR
engineering I-DIPLOMA_MAJOR
or O
computer B-DIPLOMA_MAJOR
engineering I-DIPLOMA_MAJOR
. O
5+ B-EXPERIENCE
years I-EXPERIENCE
of I-EXPERIENCE
industry I-EXPERIENCE
experience I-EXPERIENCE
. I-EXPERIENCE
Familiar O
with O
storage B-SKILLS
server I-SKILLS
architectures I-SKILLS
with O
HDD B-SKILLS

In order to convert from IOB to JSON (see documentation here), we use spaCy 3 command:

!python -m spacy convert drive/MyDrive/train_set_bert.tsv ./ -t json -n 1 -c iob
!python -m spacy convert drive/MyDrive/dev_set_bert.tsv ./ -t json -n 1 -c iob

After conversion to spaCy 3 JSON, we need to convert both the training and dev JSON files to .spacy binary file using this command (update the file path with your own):

!python -m spacy convert drive/MyDrive/train_set_bert.json ./ -t spacy!python -m spacy convert drive/MyDrive/dev_set_bert.json ./ -t spacy

• Open a new Google Colab project and make sure to select GPU as hardware accelerator in the notebook settings.
• In order to accelerate the training process, we need to run parallel processing on our GPU. To this end we install the NVIDIA 9.2 cuda library:

!wget https://developer.nvidia.com/compute/cuda/9.2/Prod/local_installers... -O cuda-repo-ubuntu1604–9–2-local_9.2.88–1_amd64.deb!dpkg -i cuda-repo-ubuntu1604–9–2-local_9.2.88–1_amd64.deb!apt-key add /var/cuda-repo-9–2-local/7fa2af80.pub!apt-get update!apt-get install cuda-9.2

To check the correct cuda compiler is installed, run: !nvcc –version

• Install the spacy library and spacy transformer pipeline:

pip install -U spacy
!python -m spacy download en_core_web_trf

• Next, we install the pytorch machine learning library that is configured for cuda 9.2:

pip install torch==1.7.1+cu92 torchvision==0.8.2+cu92 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

• After pytorch install, we need to install spacy transformers tuned for cuda 9.2 and change the CUDA_PATH and LD_LIBRARY_PATH as below. Finally, install the cupy library which is the equivalent of numpy library but for GPU:

!pip install -U spacy[cuda92,transformers]
!export CUDA_PATH=”/usr/local/cuda-9.2"
!export LD_LIBRARY_PATH=$CUDA_PATH/lib64:$LD_LIBRARY_PATH
!pip install cupy

• SpaCy 3 uses a config file config.cfg that contains all the model training components to train the model. In spaCy training page, you can select the language of the model (English in this tutorial), the component (NER) and hardware (GPU) to use and download the config file template.

Spacy 3 config file for training. Source

The only thing we need to do is to fill out the path for the train and dev .spacy files. Once done, we upload the file to Google Colab.

• Now we need to auto-fill the config file with the rest of the parameters that the BERT model will need; all you have to do is run this command:

!python -m spacy init fill-config drive/MyDrive/config.cfg drive/MyDrive/config_spacy.cfg

I suggest to debug your config file in case there is an error:

!python -m spacy debug data drive/MyDrive/config.cfg

• We are finally ready to train the BERT model! Just run this command and the training should start:

!python -m spacy train -g 0 drive/MyDrive/config.cfg — output ./

P.S: if you get the error cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed, just uninstall cupy and install it again and it should fix the issue.

If everything went correctly, you should start seeing the model scores and losses being updated:

BERT training on google colab

At the end of the training, the model will be saved under folder model-best. The model scores are located in meta.json file inside the model-best folder:

“performance”:{“ents_per_type”:{“DIPLOMA”:{“p”:0.5584415584,“r”:0.6417910448,“f”:0.5972222222},“SKILLS”:{“p”:0.6796805679,“r”:0.6742957746,“f”:0.6769774635},“DIPLOMA_MAJOR”:{“p”:0.8666666667,“r”:0.7844827586,“f”:0.8235294118},“EXPERIENCE”:{“p”:0.4831460674,“r”:0.3233082707,“f”:0.3873873874}},“ents_f”:0.661754386,“ents_p”:0.6745350501,“ents_r”:0.6494490358,“transformer_loss”:1408.9692438675,“ner_loss”:1269.1254348834}

The scores are certainly well below a production model level because of the limited training dataset, but it’s worth checking its performance on a sample job description.

To test the model on a sample text, we need to load the model and run it on our text:

nlp = spacy.load(“./model-best”)
text = ['''Qualifications- A thorough understanding of C# and .NET Core- Knowledge of good database design and usage- An understanding of NoSQL principles- Excellent problem solving and critical thinking skills- Curious about new technologies- Experience building cloud hosted, scalable web services- Azure experience is a plusRequirements- Bachelor's degree in Computer Science or related field(Equivalent experience can substitute for earned educational qualifications)- Minimum 4 years experience with C# and .NET- Minimum 4 years overall experience in developing commercial software''']for doc in nlp.pipe(text, disable=["tagger", "parser"]): print([(ent.text, ent.label_) for ent in doc.ents])

Below are the entities extracted from our sample job description:

[("C", "SKILLS"),("#", "SKILLS"),(".NET Core", "SKILLS"),("database design", "SKILLS"),("usage", "SKILLS"),("NoSQL", "SKILLS"),("problem solving", "SKILLS"),("critical thinking", "SKILLS"),("Azure", "SKILLS"),("Bachelor", "DIPLOMA"),("'s", "DIPLOMA"),("Computer Science", "DIPLOMA_MAJOR"),("4 years experience with C# and .NET\n-", "EXPERIENCE"),("4 years overall experience in developing commercial software\n\n", "EXPERIENCE")]

Pretty impressive for only using 120 training documents! We were able to extract most of the skills, diploma, diploma major, and experience correctly.

With more training data, the model would certainly improve further and yield higher scores.

With only a few lines of code, we have successfully trained a functional NER transformer model thanks to the amazing spaCy 3 library. Go ahead and try it out on your use case and please share your results. Note, you can use UBIAI annotation tool to label your data, we offer free 14 days trial.

As always, if you have any comment, please leave a note below or email at [email protected]!

Follow us on Twitter @UBIAI5

Source Prolead brokers usa

Abstraction and Data Science — Not a great combination

Abstraction: some succinct definitions.

“Abstraction is the technique of hiding implementation by providing a layer over the functionality.”

“Abstraction, as a process, denotes the extracting of the essential details about an item, or a group of items, while ignoring the inessential details”

“Abstraction — Its main goal is to handle complexity by hiding unnecessary details from the user”

Abstraction as a concept and implementation in software engineering is good. But when extended to Data Science and overdone, becomes dangerous.

Recently, the issue of sklearn’s default L2 penalty in its logistic regression algorithm came up again.

This issue was first discovered in 2019 by Zachary Lipton.

On the same issue, an excellent blog titled ‘Scikit-learn’s Defaults are wrong’ was written by W.D. Here is the link to that article.

This article IMHO is a must read for any serious Data Scientist.

While the author of the article has excellently captured the design pattern flaws, I would just like to build on it and add the problem of ‘Too much abstraction’.

In one of my previous article, I had highlighted how abstracting away GLM in sklearn’s logistic regression makes large number of people believe that Regression in Logistic Regression is merely a misnomer and it has nothing to do with Regression!!

Below is an image from that article highlighting the issue.

So, why is ‘Too much abstraction’ a problem in Data Science?

I took the liberty of modifying Francois chollet’s famous diagram on difference between traditional programming and ML to drive home some important points regarding ‘too much abstraction’.

Firstly, in normal programming, if you do abstraction, you just abstract away the fixed rules. This works out fine in software development realm as you don’t want certain people to tinker around ‘fixed rules’ or they simply don’t care ‘how things work under the hood’.

But in Data science, if you do too much abstraction, you are also abstracting away the intuition of how the algorithm works and most importantly you are hiding away the knobs and levers necessary to tweak the model.

Let’s not forget that the role of data scientist is to develop intuition of how the algorithms works and then tweak the relevant knobs/ levers to make the model a right fit to solve business problem.

Taking this away from Data Scientists is just counter intuitive.

These aside, there are other pertinent questions on ‘too much abstraction’.

Let’s revisit one of the Abstraction definitions from above: “Abstraction — Its main goal is to handle complexity by hiding unnecessary details from the user.”

When it comes to data science libraries or low code solutions, the question arises who decides ‘what is unnecessary’? Who decides which knobs and levers a user can or can’t see and tweak?

Are the people making these decisions well trained in Statistics and machine learning concepts? or are they coming from a purely programming background?

In this regard I can’t help but loan some apt excerpts from W.D ‘s article “One of the more common concerns you’ll hear–not only from formally trained statisticians, but also DS and ML practitioners–is that many people being churned through boot camps and other CS/DS programs respect neither statistics nor general good practices for data management”.

On the user side in Data Science, here are the perils of using libraries or low code solutions with ‘too much abstraction’.

• Nobody knows the statistical/ML knowledge level of the user or the training they may or may not have had.
• At the hands of a person with poor stats/ML knowledge these are just ‘execute the lines with closed eyes’ and see the magic happen.

The dangers of doing Data Science wrongly just becomes that much exacerbated. Not to mention ‘You don’t need math for ML’ and ‘Try all models’ kind of articles encouraging people to do data science without much diligence. Any guesses for what could go wrong ?

Data science is not some poem that it can be interpreted in any which way. There is a definitive right and wrong way to do data science and implement data science solutions.

Also, Data Science is just not about predictions. How these predictions are made and what ingredients led to those predictions also matter a lot. ‘Too much abstraction’ abstracts out these important parts too.

Read the Documentation

Coming to defense of these ‘too much abstracted’ libraries and solutions, some remark the user should ‘Read the documentation carefully and in detail’.

Well not many have the time and most importantly some low code solutions and libraries are sold on the idea of ‘Perform ML in 2–3 lines of code’ or ‘Do modelling faster’.

So again, referencing W.D, ‘read the doc is a cop-out’. Especially if it comes from low code solution providers.

A Bigger Problem to Ponder Upon

Having said all this, Sklearn is still by and large a good library for Machine Learning. The problem of L2 default might be one of the very few flaws.

However, I would urge the readers to ponder over this:

If abstracting away some details in one ML algorithm could cause so much issues, imagine what abstracting away details from dozen or so ML algorithms in a single line of code could result in. Some low code libraries do exactly that.

I am not against abstraction or automation per say. My concern is only with ‘too much abstraction’ . And I don’t have a concrete answer for how to tackle ‘too much abstraction’ in data science libraries. One can only wonder if there is even a middle ground.

But one thing is very clear. The issues of ‘too much abstraction’ in Data Science are real.

The more one abstracts away, the more is the chance of doing data science wrongly.

Perhaps all we can do is, be wary of low code solutions and libraries. Caveat emptor.

Source Prolead brokers usa

What is Data Mesh?

Data mesh is an architectural paradigm that unveils analytical data at scale, rapidly releasing access to an increasing number of distributed domain data sets for a proliferation of consumption scenarios such as machine learning, analytics, or data-intensive applications across the organization. It addresses the standard failure modes of the traditional centralized data lake or data platform architecture, shifting from the centralized paradigm of a lake, or its predecessor, the data warehouse.

Data mesh shifts to a paradigm that draws from modern distributed architecture: considering domains as the first-class concern, applying platform thinking to create a self-serve data infrastructure, treating data as a product, and implementing open standardization to enable an ecosystem of interoperable distributed data products. Data Mesh acquisition needs a very high level of automation regarding infrastructure provisioning, realizing the self-service infrastructure. Every Data Product team should manage to provide what it needs autonomously.

A critical point that makes a data mesh platform successful is the federated computational governance, which provides interoperability via global standardization. The “federated computational governance” is a group of data product owners with the challenging task of making rules and simplifying the conformity to such regulations. What is decided by the “federated computational governance” should follow DevOps and Infrastructure as Code conduct.

With the help of a centralized data warehouse, data mesh solves these challenges;

• Lack of ownership
• Lack of quality: Poor data quality, thus enabling the infrastructure team to know the data they are handling
• Organizational scaling: Scaling of a business or organization, thus enabling the central team to become the center point.

Data infrastructure is the other makeup of a data mesh. Data infrastructure entails the provision of access control to data, its storage, a pipeline, and a data catalog. The main goal of the data infrastructure is to avert any duplication of data in an organization. Every data product team focuses on building its own data products faster and independently. This way, the data infrastructure platform is compatible with different data domain types.

Why use a data mesh?

Allowing greater autonomy and flexibility for data owners, facilitating greater data experimentation and innovation while lessening the burden on data teams to field the needs of every data consumer through a single pipeline.

Data meshes’ self-serve infrastructure-as-a-platform provides data teams with a universal, domain-agnostic, and often automated approach to data standardization, data product lineage, data product monitoring, alerting, logging, and data quality metrics.

Provides a competitive edge compared to traditional data architectures, which are often hamstrung by the lack of data standardization between investors and consumers.

Conclusion

A data mesh helps the organization to escape the analytical and consumptive confines of monolithic data architectures and connects siloed data. To enable ML and automated analytics at scale. The data mesh allows the company to be data-driven and give up data lakes and data warehouses. It replaces them with the power of data access, control, and connectivity. If you want to know more, reach us at Dqlabs.ai, and we’ll be glad to get answers to all your queries.

Source Prolead brokers usa