COVID-19 Drives AI in Medical Imaging

Source Prolead brokers usa

A Simple Regression Problem

This article is part of a new series featuring problems with solution, to help you hone your machine learning and pattern recognition skills. Try to solve this problem by yourself first, before looking at the solution. Today’s problem also has an intriguing mathematical appeal and solution: this allows you to check if your solution found using machine learning techniques, is correct or not. The level is for beginners. 

The problem is as follows. Let X1, X2, X3 and so on be a sequence recursively defined by Xn+1 = Stdev(X1, …, Xn). Here X1, the initial condition, is a positive real number or random variable. Thus,

It is clear that Xn = An X1, where An is a number that does not depend on X1. So we can assume, without loss of generality, that X1 = 1. For instance, A1 = 1 and A2 = 0. The purpose here is to study the behavior of An (for large n) using simple model fitting techniques. I plotted the first few values of An, below. In the figure below, the X-axis represents n, and the Y-axis represents An. The question is: how to approximate An as a simple function of n? Of course, a linear regression won’t work. What about a polynomial regression?

The first 600 values of An are available here, as a text file.

Solution

A tool as basic as Excel is good enough to find the solution. However, if you use Excel, the built-in function Stdev has a correcting factor that needs to be taken care of. But you can just use the values of An available in my text file mentioned above, to avoid this problem.

If you use Excel, you can try various types of trend lines to approximate the blue curve, and even compute the regression coefficients and the R-squared for each tested model. You will find very quickly that the power trend line is the best model by far, that is, An is very well approximated (for large values of n) by An = b n^c. Here n^c stands for n at power c; also, b and c are the regression coefficients. In other words, log An = log b + c log n (approximately). 

What is very interesting, is that using some mathematics, you can actually compute the exact value of c. Indeed, c is solution of the equation c^2 = (2c + 1) (c + 1)^2, see here. This is a polynomial equation of degree 3, so the exact value of c can be computed. The approximation is c = -0.3522011. It is however very hard to get the exact value of b. 

It would interesting to plot the residual error for each estimated value of An, and see if it shows some pattern. This could lead to a better approximation: An = b n^c (1 + d / n), with three parameters: b, c (unchanged) and d.

Source Prolead brokers usa

DSC Weekly Digest 27 July 2021

There are fashions in technology that are every bit as ephemeral as fashions in the garment industry. For a while, all data was BIG DATA, then data warehouses were cool, then data lakes became the gotta-have look for the year. Data science had its heyday, and everyone had to stock up on PhDs, then knowledge graphs gained a brief bit of currency, like a particularly frilly collar or gold chains. DevOps was hot and everyone wanted to be a DevOps tech, then machine learning was hot and everyone became a machine learning guru. Yesterday we were arguing about whether R or Python was the next big thing, and today it’s shifted to AutoMLOps vs. AIOps. 

Everyone is currently chasing the holy grail of being data-driven companies, often with at best only a very faint idea about what that actually means. Every so often, it is worth stepping off the carousel and letting the brass ring go past,

In general, data can be thought of as records of the events that take place around a person or an organization as they take place Some of this information is a record of the events themselves, such as sales transactions. Some of the data is contextual metadata that puts the events into perspective.

It’s worth noting that some of this data has no relevance to you or your organization, which we refer to as noise, while other data does have relevance, which can be referred to as signal. Unfortunately, there is no explicit guide about what is noise and what is signal until you have a question or query to ask, and typically the biggest problem that most organizations face is that they tend to hold on to transactional data preferentially to metadata, despite the fact that it is frequently the latter that holds the answer to the queries, simply because transactional data is easiest to capture.

Data analytics, at its core, is the art of knowing how to ask the right questions. Not surprisingly, data analytics is stochastic or probabilistic in nature because it is based upon the assumption that people and organizations that act a certain way in the past will continue to do so into the future. This is true, so long as the conditions that applied in the past also continue into the future, and because people’s behaviors have a certain degree of momentum, it is even somewhat true when the conditions change, at least for a little while. However, the future is notoriously fuzzy around inflection points, where events change in dramatic ways, and in those times a good data scientist is worth their Ph.D.-enhanced salaries.

A data-driven organization then is one that both practices good “data hygiene” in the acquisition and preparation of data (typically by attempting to determine semantics or meaning in that data independent of the form of that data) as well as utilizes that data in order to not only read the tea leaves but also to change the behavior of that organization in response to changes in data. Failure to change when the model indicates that change is warranted makes everything else that happens in the data process moot – it is an exercise in adding process without using that process for something positive.

In many respects, the goal of being data-driven, then, is to make the organization become aware in the same way that an animal is aware of its surroundings and can react when those surroundings change, or the way that a seasoned captain aboard a sailing ship can read the sky and know whether to unfurl the sails to catch favorable winds or to furl them to protect the ship from storms.

A data-driven organization is one that is capable of discerning the signal from the noise and acting in response. All else is marketing.

In media res,

Kurt Cagle
Community Editor,
Data Science Central

To subscribe to the DSC Newsletter, go to Data Science Central and become a member today. It’s free! 

Source Prolead brokers usa

The Tao of Tau

class Tau {     

        static TAU = 2 * Math.PI

        static TAU_2 = this.TAU/2

        static TAU_4 = this.TAU/4

        static TAU_6 = this.TAU/6

        static TAU_8 = this.TAU/8

        static TAU_12 = this.TAU/12

        constructor(){

            // This consists solely of static methods and constants

        }

        // converts a revolution to degrees

        static toDegrees(rev){return rev * 360}

        // converts a revolution to radians

        static toRadians(rev){return rev * this.TAU}

        // converts from degrees to revolutions

        static fromDegrees(deg){return deg / 360}

        // converts from radians to revolutions

        static fromRadians(rad){return rad / this.TAU}

        // returns the sine value of the given revolution

        static sin(rev){

            return Math.sin(rev * this.TAU)

        }

        // returns the cosine value of the given revolution

        static cos(rev){

            return Math.cos(rev * this.TAU)

        }

        // returns the tangent value of the given revolution

        static tan(rev){

            return Math.tan(rev * this.TAU)

        }

        // returns the arcsine value of the given revolution

        static asin(rev){

            return this.fromRadians(Math.asin(rev))

        }

        // returns the arccosine value of the given revolution

        static acos(rev){

            return this.fromRadians(Math.acos(rev))

        }

        // For a given x,y value, returns the corresponding revolution from -0.5 to 0.5.

        static atan(x,y){

            return this.fromRadians(Math.atan2(y,x))    }

    

    }

    

    class TauComplex{

        // Indicates the number of significant digits complex numbers are displayed using.

        static SIGDIGITS = 5;

        constructor(x,y){

            this.x = x

            this.y = y

            return this

        }

        // toString() generates a complex number of the form “a+bi” for string output

        toString(){

            let minX = Math.abs(this.x)/span>1e-5?0:TauComplex.trim(this.x);

            let minY = Math.abs(this.y)/span>1e-5?0:TauComplex.trim(this.y);

            return `${minX} ${Math.sign(this.y)>=0?‘+’:‘-‘} ${Math.abs(minY)}i`

        }

        // generates the length of the complex number vector

        get modulus(){

            return Math.sqrt(this.x*this.x + this.y*this.y);

        }

        // generates the square of the length of the complex number vector. This avoids the need to take the square root

        get modsquare(){

            return this.x*this.x + this.y*this.y;

        }

        // retrieves the angle relative to the positive x axis of the complex number, in revolutions

        get theta(){

            let angle = Tau.atan(this.x,this.y);

            let ySgn = Math.sign(this.y);

            let adjAngle = ySgn/span>0?1+angle:angle;

            return adjAngle;

        }

        // retrieves the complex conjugate (a-bi) of the complex number (a+bi)

        get conjugate(){

            return new TauComplex(this.x,-this.y)

        }

        // retrieves the complex inverse of the number (a+bi).

        get inverse(){

            return (this.conjugate).scale(1/this.modsquare)

        }

        // rotates the complex number through the angle, expressed in revolutions.

        rotate(angle){

            let newX = this.x * Tau.cos(angle) – this.y * Tau.sin(angle);

            let newY = this.x * Tau.sin(angle) + this.y * Tau.cos(angle)

            return new TauComplex(newX,newY)

        }

        // Multiplies the complex number by a scalar value (or values if two arguments are supplied)

        scale(x,y=x){

            let newX = this.x * x;

            let newY = this.y * y;

            return new TauComplex(newX,newY)

        }

        // translates the complex number by the given amount. Equivalent to adding two complex numbers

        translate(x,y=x){

            let newX = this.x + x;

            let newY = this.y + y;

            return new TauComplex(newX,newY)

        }

        // Adds two or more complex numbers together.

        static sum(…c){

            let reducer = (acc, cur) => new TauComplex(acc.x+cur.x,acc.y+cur.y)

            return c.reduce(reducer)

        }

        // Multiples two or more complex numbers together.

        static mult(…c){

            let reducer = (acc, cur) => new TauComplex(acc.x*cur.x–acc.y*cur.y,acc.x*cur.y+acc.y*cur.x)

            return c.reduce(reducer)

        }

        // Divides the first complex number by the second

        static div(c1,c2){

            return TauComplex.mult(c1,c2.inverse)

        }

        // Takes the complex number to the given power. Power MUST be a non-negative integer.

        pow(power){

            let arr = [];

            for (var index=0;index!=power;index++){

                arr.push(this);

            }

            if (arr.length>0) {

                return TauComplex.mult(…arr)

            }

            else {

                return new TauComplex(1,0);

            }

        }

        // Returns the real portion of a complex number

        get re(){

            return this.x

        }

        // Returns the imaginary portion of a complex number

        get im(){

            return this.y

        }

        // Returns the complex number associated with a unit vector rotated by the revolution amount

        static tau(rev){

            return new TauComplex(Tau.cos(rev),Tau.sin(rev));

        }

        // Returns the complex exponent of the given complex number

        get exp(){

            return TauComplex.tau(this.y).scale(Math.exp(this.x))

        }

        // Creates a string representation of a number to the given significant digits, default being 5.

        static trim(value,sigDigits=this.SIGDIGITS){

            return value.toLocaleString(“en-us”,{maximumSignificantDigits:sigDigits})

        }

        static array(…arr){

            return arr.map((subArr,index)=>new TauComplex(…subArr))

        }

    }

    const _TauComplex = TauComplex;

    exports.TauComplex = _TauComplex;

    const _Tau = Tau;

    exports.Tau = _Tau;

Source Prolead brokers usa

Seven Ways that AI in Telecom is Transforming the Enterprises

The telecom sector is no longer limited to delivering basic phone and internet service. In the Internet of Things (IoT) era, mobile and broadband services are driving technological innovation. Telcos are utilizing AI to handle and analyze these massive amounts of Big Data to extract meaningful insights and improve customer experience, operations, and revenue through new products and services. With Gartner predicting that 20.4 billion connected devices would be in use globally by 2020, more service providers can see the benefit of AI in the telecom industry, including optimization, maintenance, client targeting, and more.

1. Network Optimization

To strengthen their infrastructure, 63.5 percent of operators are investing in AI systems. In the telecom industry, artificial intelligence is critical for CSPs to develop self-optimizing networks (SONs), which allow operators to autonomously adjust network quality based on traffic data by region and time zone. Artificial intelligence in the telecom industry utilizes powerful algorithms to seek patterns in data. It allows telcos to discover, forecast network anomalies, and proactively address problems before customers are harmed.

2. Detecting and preventing fraud

ML algorithms cut down fraudulent activities happening in the telecom industry, such as fake profiles, illegal access, etc. With the aid of advanced ML algorithms, the system can detect the irregularities occurring on a real-time basis, which is more effective than what human analysts can perform.

3. Enabling predictive analytics

By combining data, complex algorithms, and machine learning approaches to anticipate future results based on historical data, AI-driven predictive analytics assists telcos in providing better services. It means that operators may use data-driven insights to track the health of equipment, predict failure based on patterns, and prevent problems with communications hardware like cell towers, power lines, data center servers, and even set-top boxes in consumers’ homes. Through the process of Predictive Analytics, CSPs can make efficient and effective business decisions. Technologies such as network automation and intelligence will allow for more accurate root cause investigation and issue prediction. With technologies such as AI/ML, it will support more strategic aims like creating new consumer experiences and efficiently dealing with evolving company needs.

4. Optimizing Service Quality

Machine learning and artificial intelligence in telecom can assist you in improving the quality of your service. You can apply Machine Learning techniques to forecast how your network’s consumption will change over time across the different geographies it serves. In order to improve optimization, a variety of criteria can be considered, including time zone, hour, weather, national or regional holidays, and more.

5. Improve Customer Service

Another advantage of AI in telecom is through the automation of the customer service mechanism. It can help telcos reinvent customer relationships through personalized, intelligent, and persistent two-way conversations at scale. Conversational AI systems are another use-case of AI in telecom. According to Juniper Research, virtual assistants have learned to automate and scale one-on-one conversations so effectively that they are expected to save businesses up to $8 billion annually by 2022. The large volume of support requests for installation, set up, troubleshooting, and maintenance, which often overwhelm customer service centers, has led telcos to turn to virtual assistants for assistance. Operators can add self-service capabilities that show customers how to install and run their own devices using data science, AI, and machine learning.

6. Preventing Malicious Activity

Machine Learning can effectively protect your network from dangerous behaviours such as DDoS attacks. Using AI in telecom, the network can be trained to recognize a large number of similar requests that are inundating it.  At the same time, it lets them decide whether to deny these requests outright or shunt them to a less busy data center to be handled manually by your staff.

7. Foster innovation and drive new business

One of the promises of 5G is to bring Industry 4.0 use cases to fruition by enabling high speed, low latency, and dense deployment of endpoints such as sensors, robots, and video cameras. It opens up new business opportunities for telcos to not only outsourced IT services to the enterprises but also offer innovative services driven by AI at the Edge. New innovative AI-driven services are geared to address many new business segments, for which telecom operators will be one of the beneficiaries.

AI technology plays an essential role in digital transformation across all industries and verticals. The crucial integration of AI in the telecom industry will help assist and guide CSPs in delivering, managing, and optimizing the telecom infrastructure and networks.

Do you agree artificial intelligence is changing the telecom industry? If yes, how will it benefit an operator? Feel free to share your thoughts in the comments section.

Learn how augmented analytics can help transform your approach to e…

Source Prolead brokers usa

Transforming Retail Industry with AI-Based Personalized Customer Experience

Artificial Intelligence and data analytics solutions have been the driving force behind the growth of various industries. Manufacturing, healthcare, finance, and most importantly, the retail sector are business verticals leveraging these innovative technologies’ potential.

Among the various industries, the retail sector is massively implementing AI and data analytics solutions. The industry is witnessing this accelerated growth in implementing these technologies due to reduced customer churn, improvement of customer retention rate, and, most importantly, to offer a better customer experience.

Retailers across the globe have now understood that customer insights can not just add profit to their business, but they can also help them add value to their business in terms of customer satisfaction, higher retention, and improved customer acquisition. As a result, the retailers implement technologies and services such as AI, advanced data analytics, and machine learning to their businesses to collect, process, and visualize data to generate actionable insights.

According to a report by Research and Market, the global retail analytics market is projected to grow with a 19.4% CAGR. The report also states that the retail analytics market shall reach a value of US$ 10.4 billion by the end of 2023.

Now that the rate at which retailers are implementing technologies like data analytics and artificial intelligence into their business is known let’s figure out the impact of these technologies in developing actionable customer insights.

1. Strategy

The first and foremost step to generate the insight is to develop an effective strategy to collect the data from sources such as comments, reviews, shopping patterns, and products purchased. It is highly recommended to create a roadmap that will allow the retailers to collect, process, and use the data to develop personalized experiences. With the help of data analytics assessment and strategy services, retailers can develop a plan of action that shall help them attract new customers while retaining the existing ones.

2. Marketing

To improve the reach of the business, effective marketing plans are a mandatory asset for any retailer. Data analytics solutions such as customer segmentation, data mining, and customer value analysis allow retailers to understand their customers’ preferences, frequent purchases, and purchases. This will enable them to provide preference-based offers and schemes, improving the customers’ overall shopping experience.

3. Customer Relationship

Retaining customers purely depends on the relationship a retailer has with its visitors. Providing appealing offers and personalized discounts may be one of the parameters in strengthening the relationship. However, the majority of customer relationships happen with after-sales services. How well a retailer provides maintenance of the product? And, How quickly the customer receives support from the retailer? Nevertheless, the major challenge associated with customer relationships is the retailers’ lack of maintenance and support. As a result, the retailer has to face customer complaints, churns, and loss of prospects.

Source Prolead brokers usa

Twenty Projects in Data Science Using Python (Part-I)

Young and dynamic data science and machine learning enthusiasts are all are very interested in making a career transition by learning and doing as much hands-on learning as possible with these technologies and concepts as Data Scientists or Machine Learning Engineers or Data Engineers or Data Analytics Engineers. I believe they must have the Project Experience and a job-winning portfolio in hand before they hit the interview process.

Certainly, this interview process would be challenging, NOT only for the freshers, but also for experienced individuals since these are all new techniques, domain, process approach, and implementation methodologies that are totally different from traditional software development. Of course, we could adopt an agile mode of delivery and no excuse from modern cloud adoption techniques and state beyond all industries and domains, who are all looking and interested in artificial intelligence and machine learning (AI and ML) and its potential benefits.

In this article, let’s discuss how to choose the best data science and ML projects during the capstone stages of your schools, colleges, training institutions, and specific job-hunting perspective. You could map this effort with our journey towards getting your dream job in the data science and machine learning industry.

Without much ado, here are the top 20 machine learning projects that can help you get started in your career as a machine learning engineer or data scientist. Let us move into a curated list of data science and machine learning projects for practice that can be a great add-on to your portfolio –

1. Data Science Project – Ultrasound Nerve Segmentation

Problem Statement & Solution

In this project, you will be working on building a machine learning model that can identify nerve structures in a data set of ultrasound images of the neck. This will help enhance catheter placement and contribute to a more pain-free future.

Even the bravest patients cringe at the mention of a surgical procedure. Surgery inevitably brings discomfort, and oftentimes involves significant post-surgical pain. Currently, patient pain is frequently managed using narcotics that bring a number of unwanted side effects.

This data science project’s sponsor is working to improve the pain management system using indwelling catheters that block or mitigate pain at the source. These pain management catheters reduce dependence on narcotics and speed up patient recovery.

The project objective is to precisely identify the nerve structures in the given ultrasound images, and this is a critical step in effectively inserting a patient’s pain management catheter. This project has been developed in python language, so it is easy to understand the flow of the project and the objectives. They must build a model that can identify nerve structures in a dataset of given ultrasound images of the neck. Doing so would improve catheter placement and contribute to a more pain-free future.

Let see the simple workflow.

Certainly, this project would help us to understand the image classification and highly sensitive area of analysis in the medical domain.

Take away and outcome and of this project experience.

• Understanding what image segmentation is.
• Understanding of subjective segmentation and objective segmentation
• The idea of converting images into matrix format.
• How to calculate euclidean distance.
• Scope of what dendrogram are and what they represent.
• Overview of agglomerative clustering and its significance
• Knowledge of VQmeans clustering
• Experiencing grayscale conversion and reading image files.
• A practical way of converting masked images into suitable colours.
• How to extract the features from the images.
• Recursively splitting a tile of an image into different quadrants.

2. Machine Learning project for Retail Price Optimization

Problem Statement & Solution

In this machine learning pricing project, we must implement retail price optimization and apply a regression trees algorithm. This is one of the best ways to build a dynamic pricing model, so developers can understand how to build models dynamically with commercial data which is available from a nearby source and visualization of the solution is tangible.

In this competitive business world “PRICING A PRODUCT” is a crucial aspect. So, we must gather a lot of thought process into that solution approach. There are different strategies to optimize the pricing of products. And must take extra care during the pricing of the products due to their sensitive impact on the sales and forecast. While there are products whose sales are not very affected by their price changes, they could be luxury items or essentials products in the market. This machine learning retail price optimization project will focus on the former type of products.

This project clearly captures the data and aligns with the “Price Elasticity of Demand” phenomenon. This exposes the degree to which the effective desire for something changes as its price the customers desire could drop sharply even with a little price increase, I mean directly proportional relationship. Generally, economists use the term elasticity to denote this sensitivity to price increases.

In this Machine Learning Pricing Optimization project, we will take the data from the café shop and, based on their past sales, identify the optimal prices for their list of items, based on the price elasticity model of the items. For each café item, the “Price Elasticity” will be calculated from the available data and then the optimal price will be calculated. A similar kind of work can be extended to price any products in the market. 

Take away and Outcome and of this project experience.

• Understanding the retail price optimization problem
• Understanding of price elasticity (Price Elasticity of Demand)
• Understanding the data and feature correlations with the help of visualizations
• Understanding real-time business context with EDA (Exploratory Data Analysis) process
• How to segregate data based on analysis.
• Coding techniques to identify price elasticity of items on the shelf and price optimization.

3. Demand prediction of driver availability using multistep Time Series Analysis

Problem Statement & Situation

In this supervised learning machine learning project, you will predict the availability of a driver in a specific area by using multi-step time series analysis. This project is an interesting one since it is based on a real-time scenario.

We all love to order food online and do not like to experience delivery fee price variation. Delivery charges are always highly dependent on the availability of drivers in your area in and around, so the demand of orders in your area, and distance covered would greatly impact the delivery charges. Due to driver unavailability, there is an impact in delivery pricing increasing and directly this will hit the many customers who have dropped off from ordering or moving into another food delivery provider, so at the end of the day food suppliers (Small/medium scale restaurants) are reducing their online orders.

 To handle this situation, we must track the number of hours a particular delivery driver is active online and where he is working and delivering foods, and how many orders are in that area, so based on all these factors certainly, we can efficiently allocate a defined number of drivers to a particular area depending on demand as mentioned earlier.

Take away and Outcome and of this project experience.

• How to convert a Time Series problem to a Supervised Learning problem.
• What exactly is Multi-Step Time Series Forecast analysis?
• How does Data Pre-processing function in Time Series analysis?
• How to do Exploratory Data Analysis (EDA) on Time-Series?
• How to do Feature Engineering in Time Series by breaking Time Features to days of the week, weekends.
• Understand the concept of Lead-Lag and Rolling Mean.
• Clarity of Auto-Correlation Function (ACF) and Partial Auto-Correlation Function (PACF) in Time Series.
• Different strategic approaches to solving Multi-Step Time Series problem
• Solving Time-Series with a Regressor Model
• How to implement Online Hours Prediction with Ensemble Models (Random Forest and Xgboost)

4. Customer Market Basket Analysis using Apriori and FP- growth algorithms

Problem Statement & Solution

In this project, anyone can learn how to perform Market Basket Analysis (MBA) with the application of Apriori and FP growth algorithms based on the concept of association rule learning, one of my favourite topics in data science. 

Mix and Match is a familiar term in the US, I remember I used to get the toys for my kid. It was the ultimate experience you know. Same time keeping things together nearby, like bread and jam–shaving razor and cream, these are the simple examples for MBA, and this is making the customer buy additional purchases more likely.

It is a widely used technique to identify the best possible mix of products or services that comes together commonly. This is also called “Product Association Analysis” or “Association Rules”. This approach is best fit physical retail stores and even online too. In other ways, it can help in floor planning and placement of products.

5. E-commerce product reviews – Pairwise ranking and sentiment analysis.

Problem Statement & Solution

Product recommendation systems for the products which are sold over the online-based pairwise ranking and sentiment analysis. So, we are going to perform sentiment analysis on product reviews given by the customers who are all purchased the items and ranking them based on weightage. Here, the reviews play a vital role in product recommendation systems.

Obviously, reviews from customers are very useful and impactful for customers who are going to buy the products. Generally, a huge number of reviews in the bucket would create unnecessary confusion in the selection and buying interest on a specific product. If we have appropriate filters from the collective informative reviews. This proportional issue has been attempted and addressed in this project solution.

This recommendation work has been done in four phases.

• Data pre-processing/filtering
  • Which includes.
    • Language Detection
    • Gibberish Detection
    • Profanity Detection
  • Feature extraction,
  • Pairwise Review Ranking, 

The outcome of the model will be a collection of the reviews for a particular product and its ranking based on relevance using a pairwise ranking approach method/model.

Take away and Outcome and of this project experience.

• EDA Process
  • Over Textual Data
  • Extracted Featured with Target Class
• Using Featuring Engineering and extracting relevance from data
• Reviews Text Data Pre-processing in terms of
  • Language Detection
  • Gibberish Detection
  • Profanity Detection, and Spelling Correction
• Understand how to find gibberish by Markov Chain Concept
• Hands-On experience on Sentiment Analysis
  • Finding Polarity and Subjectivity from Reviews
• Learning How to Rank – Like Pairwise Ranking
• How to convert Ranking into Classification Problem
• Pairwise Ranking reviews with Random Forest Classifier
• Understand the Evaluation Metrics concepts
  • Classification Accuracy and Ranking Accuracy

6. Customer Churn Prediction Analysis using Ensemble Techniques

Problem Statement & Solution

In some situations, the customers are closing their accounts or switching to other competitor banks for too many reasons. This could cause a huge dip in their quarterly revenues and might significantly affect annual revenues for the enduring financial year, this would directly cause the stocks to plunge and the market cap to reduce considerably. Here, the idea is to be able to predict which customers are going to churn, and how to retain them, with necessary actions/steps/interventions by the bank proactively.

In this project, we must implement a churn prediction model using ensemble techniques.

Here we are collecting customer data about his/her past transactions details with the bank and statistical characteristics information for deep analysis of the customers. With help of these data points, we could establish relations and associations between data features and customer’s tendency to possible churn. Based on that, we will build a classification model to predict whether the specific set of customers(s) will indeed leave the bank or not. Clearly draw the insight and identify which factor(s) are accountable for the churn of the customers.

Take away and Outcome and of this project experience.

• Defining and deriving the relevant metrics
• Exploratory Data Analysis
  • Univariate, Bivariate analysis,
  • Outlier treatment
  • Label Encoder/One Hot Encoder
• How to avoid data leakage during the data processing
• Understanding Feature transforms, engineering, and selection
• Hands-on Tree visualizations and SHAP and Class imbalance techniques
• Knowledge in Hyperparameter tuning
  • Random Search
  • Grid Search
• Assembling multiple models and error analysis.

7. Build a Music Recommendation Algorithm using KKBox’s Dataset.

Problem Statement & Solution 

Music Recommendation Project using Machine Learning to predict the best chances of a user listening and loving a song again after their very first noticeable listening event. As we know, the most popular evergreen entertainment is music, no doubt about that. There might be a mode of listening on different platforms, but ultimately everyone will be listening to music with this well-developed digital world era.  Nowadays, the accessibility of music services has been increasing exponentially ranging from classical, jazz, pop etc.,

Due to the increasing number of songs of all genres, it has become very difficult to recommend appropriate songs to music lovers. The question is that the music recommendation system should understand the music lover’s favourites and inclinations to other similar music lovers and offer the songs to them on the go, by reading their pulse.

In the digital market, we have excellent music streaming applications available like YouTube, Amazon Music, Spotify etc., All they have their own features to recommend music to music lovers based on their listening history and first and best choice. This plays a vital role in this business to catch the customers on the go. Those recommendations are used to predict and indicate an appropriate list of songs based on the characteristics of the music, which has been heard by music lovers over the period.

This project uses the KKBOX dataset and demonstrates the machine learning techniques that can be applied to recommend songs to music lovers based on their listening patterns which were created from their history.

Take away and Outcome and of this project experience.

• Understanding inferences about data and data visualization
• Gaining knowledge on Feature Engineering and Outlier treatment
• The reason behind Train and Test split for model validation
• Best Understanding and Building capabilities on the algorithm below
  • Logistic Regression model
  • Decision Tree classifier
  • Random Forest Classifier
  • XGBoost model

8.Image Segmentation using Masked R-CNN with TensorFlow

Problem Statement & Solution

Fire is one of the deadliest risk situations. Generally, fire can destroy an area completely in a very short span of time. Another end this leads to an increase in air pollution and directly affects the environment and an increase in global warming. This leads to the loss of expensive property. Hence early fire detection is very important.

The Object of this project is to build a deep neural network model that will give precise accuracy in the detection of fire in the given set of images. In this Deep Learning-based project on Image Segmentation using Python language, we are going to implement the Mask R-CNN model for early fire detection.

In this project, we are going to build early fire detection using the image segmentation technique with the help of the MRCNN model. Here, fire detection by adopting the RGB model (Color: Red, Green, Blue), which is based on chromatic and disorder measurement for extracting fire pixels and smoke pixels from the image. With the help of this model, we can locate the position where the fire is present, and which will help the fire authorities to take appropriate actions to prevent any kind of loss.

Take away and Outcome and of this project experience.

• Understanding the concepts
  • Image detection
  • Image localization
  • Image segmentation
  • Backbone
    • Role of the backbone (restnet101) in Mask RCNN model
  • MS COCO
• Understanding the concepts
  • Region Proposal Network (RPN)
  • ROI Classifier and bounding box Regressor.
• Distinguishing between Transfer Learning and Machine Learning.
• Demonstrating image annotation using VGG Annotator.
• The best understanding of how to create and store the log files per epoch.

9. Loan Eligibility Prediction using Gradient Boosting Classifier

Problem Statement & Solution

In this project, we are predicting if a loan should be given to an applicant or not for the given data of various customers who are all seeking the loan based on several factors like their credit score and history. The ultimate aim is to avoid manual efforts and give approval with the help of a machine learning model, after analyzing the data and processing for machine learning operations. On the top of the machine, the learning solution will look at different factors based on testing the dataset and decide whether to grant a loan or not to the respective individual.

In this ML problem, we use to cleanse the data and fill in the missing values and bringing various factors of the applicant like credit score, history and from those we will try to predict the loan granting by building a classification model and the output will be giving output in the form of probability score along with Loan Granted or Refused as output from the model.

Take away and Outcome and of this project experience.

• Understanding in-depth:
  • Data preparation
  • Data Cleansing and Preparation
  • Exploratory Data Analysis
  • Feature engineering
  • Cross-Validation
  • ROC Curve, MCC scorer etc
  • Data Balancing using SMOTE.
  • Scheduling ML jobs for automation
• How to create custom functions for machine learning models
• Defining an approach to solve
  • ML Classification problems
  • Gradient Boosting, XGBoost etc

10.Human Activity Recognition Using Multiclass Classification

Problem Statement & Solution

In this project we are going to classify human activity, we use multiclass classification machine learning techniques and analyze the fitness dataset from a smartphone tracker. 30 activities of daily participants have been recorded through a smartphone with embedded inertial sensors and build a strong dataset for activity recognition point of view. Target activities are WALKING, WALKING UPSTAIRS, WALKING DOWNSTAIRS, SITTING, STANDING, LAYING, by capturing 3-axial linear acceleration and 3-axial angular velocity at a constant rate of 50Hz. The objective is to classify activities mentioned above among 6 and 2 different axials. This was captured by an embedded accelerometer and gyroscope in the smartphone. The experiments have been video-recorded to label the data manually. The obtained dataset has been randomly partitioned into two sets as 70% for training and 30% for test data.

Take away and Outcome and of this project experience.

• Understanding
  • Data Science Life Cycle
  • EDA
  • Univariate and Bivariate analysis
  • Data visualizations using various charts.
  • Cleaning and preparing the data for modelling.
  • Standard Scaling and normalizing the dataset.
  • Selecting the best model and making predictions
• How to perform PCA to reduce the number of features
• Understanding how to apply
  • Logistic Regression & SVM
  • Random Forest Regressor, XGBoost and KNN
  • Deep Neural Networks
• Deep knowledge in Hyper Parameter tuning for ANN and SVM.
• How to plot the confusion matrix for visualizing the result
• Develop the Flask API for the selected model.

Project Idea Credits – ProjectPro helps professionals get their work done faster and with practical experience with verified reusable solution code, real-world project problem statements, and solutions from various industry experts.

So far, We have discussed 10 different projects, Hope you could feel each one of them at least high level and clear goal of what is the objective of the project and learning take away While doing the projects as hands-on.

I am sure you could feel the essence of those and digesting each concept in Data Science and Machine Learning. Learn More always! 

Will discuss 10 more projects in a short while, Until then, Bye! See you!

Source Prolead brokers usa

KDNuggets.com Launches Free Outlier Analysis Boxplot Template

Canadians in the civil service, academia or legal profession who use WordPerfect (Corel) and its Quattro Pro spreadsheet application should also be able to download and test the free template since it is developed on a VBA platform. You can let me know. And, anyone – anywhere – who uses a Windows-based spreadsheet application other than Excel — that is likely developed on a VBA platform — should also be able to run the free template (and others) in my store. You can let me know if this is true.

Source Prolead brokers usa

An Introduction to Statistical Sampling

Image Source: Statistical Aid

Sampling is a statistical procedure of selecting some representative part from an existing population or study area. Specifically, draw a sample from the study population using some statistical methods. For example-
if we want to calculate the average age of Bangladeshi people then we can not deal with the whole population. In that time we must have to deal with some representative part of this population. This representative part is called sample and the procedure is called sampling.

Why need sampling

—  It makes possible the study of a large population which contains different characteristics.

—  It is for economy.

—  It is for speed.

—  It is for accuracy.

—  It saves the sources of data from being all consumed.

Sometimes we can’t work with population such as blood test, in that situation sampling is must.

 

Types 

Probability Sampling

It is based on the concept of random selection where each population elements have a non-zero chance to occur as a sample. Sampling techniques can be divided into two categories: probability and non-probability. Randomization or chance is the core of probability sampling techniques.

For example, if a researcher is dealing with a population of 100 people, each person in the population would have the odds of 1 out of 100 for being chosen. This differs from non-probability sampling, in which each member of the population would not have the same odds of being selected.

Different types of probability sampling

 

Applications

·    In opinion poll, a relatively small number of persons are interviewed and their opinions on current issues are solicited in order to discover the attitude of the community as a whole.

·    At border stations, customs officers enforce the laws by checking the effects of only a small number of travelers crossing the border.

·    A departmental store wises to examine whether it is losing or gaining customers by drawing a sample from its lists of credit card holders by selecting every tenth name.

·    In a manufacturing company, a quality control officer take one sample from every lot and if any sample is damage then he reject that lot.

Advantages

—  Creates samples that are highly representative of the population.

—  Sampling bias is tens to zero.

—  Higher level of reliability of research findings.

—  Increased accuracy of sample error estimation.

—  The possibility to make inferences about the population.

Disadvantages

—  Higher complexity compared to non-probability sample.

—  More time consuming, especially when creating larger sample.

—  Usually more expensive.

Non-Probability sampling

The process of selecting a sample from a population without using statistical probability theory is called non-probability sampling.

Example

Lets say that the university has roughly 10000 students. These 10000 students are our population (N). Each of the 10000 students is known as a unit, but its hardly possible to get known and select every student randomly.

Here we can use Non-Random selection of sample to produce a result.

 

Applications

 

·   It can be used when demonstrating that a particular trait exist in the population.

·    It can also be useful when the researcher has limited budget, time and workforce.

 

Advantages

·        Select samples purposively

·        Enable researchers to reach difficult to identify members of the population.

·        Lower cost

·        Limited time.

 

Disadvantage

Difficult to make valid inference about the entire population because the sample selected is not representative.

We cannot calculate confidence interval.

Source Prolead brokers usa

Using IDP to jump-start your AI journey

At the end of 2020, Forrester Research analysts predicted that more than a third of companies would look to AI in 2021 to help with workplace disruption caused by the pandemic – that is, the shift to remote and hybrid work. This includes things like intelligent document processing (IDP) and customer service agent augmentation, among other functions.

In other words, now is the time for AI to shine – and for organizations looking to launch new AI projects, the good news is that it’s not an all-at-once proposition. Smaller applications, like those mentioned above, can be the essential first steps. IDP, in particular, can be a great entry point for organizations looking to start the artificial intelligence (AI) journey.

Dipping your toes in the AI water

With some technologies, there’s no real way to take baby steps; you have to go from zero to 100. But with AI and machine learning (ML) adoption, it really is a journey. You can try it out first with small, isolated projects, applying AI to one function at a time to see the results. It’s very much a crawl, walk, run approach. Unlike many transactional systems like ERP or CRM, AI/ML application deployment in the enterprise world is not a sudden, life-changing event. In fact, AI/ML should be adopted in a gradual manner to achieve the greatest success.

When you think about document processing, it may seem low on the priority list – one small problem in the scheme of things. But the reality is that it’s an important but often overlooked component of so many functions across the enterprise. And in that respect, yes, it’s a small piece of the bigger picture, but a key one.

And when it comes to adopting automation and AI, IDP can be a comparatively easy place to start. For a business leader who wants to start applying automation and AI within their enterprise, it represents a relatively low-risk steppingstone.

Understanding IDP

Today’s enterprises generate and receive a mountain of documents, both digital and physical. These are often manually processed by humans, who enter the relevant data into application systems for storage and future retrieval purposes. This approach is time-consuming and error-prone. It relies entirely on human efforts to process documents, which can lead to long cycle times, reduced productivity, unwanted errors and increased costs.

IDP promises to make it easier to automate these workflows through document capture, optical character recognition (OCR) and natural language processing (NLP). The premise behind IDP is to digitize the entire document processing workflow across business processes by eliminating the touchpoints that requires manual intervention. Doing away with this manual intervention not only reduces costs, but it also reduces errors and ultimately helps achieve new levels of productivity.

More specifically, IDP intelligently classifies, captures and extracts all data from documents entering the workflow. It then organizes the information based on business need. Once the data has been validated and verified, the system automatically exports it to downstream business applications. In today’s advanced IDP solutions, the entire process is powered by AI/ML algorithms to make business processes more resilient to disruptions and help mitigate risks.

This is unlike robotic process automation (RPA), which doesn’t really use AI and is mostly rule-based and driven by templates approach. It eliminates repetitive tasks but can’t provide the other benefits that IDP brings to the table.

A gateway to more business and process automation

Documents underpin so many different functions and applications, be it Accounts Payable, CRM, ERM or business process management. Being able to apply understanding and insight to integrated documents can be a huge differentiator for many other enterprise applications.

A key to the success of AI in enterprise applications is whether you believe your AI is trustworthy. Trust can be built by verification and validation. IDP provides the opportunity to easily verify and validate whether AI is doing what it is supposed to be doing. This makes it easy for enterprises to adapt AI for other key business applications once trust has been established.

All of this fits into the bigger picture of meeting the aforementioned goals – cutting costs, reducing time spent on manual tasks, reducing risk of human error and increasing productivity – throughout the enterprise.

The journey begins

To paraphrase Ralph Waldo Emerson, when it comes to AI/ML adoption, it’s a journey, not a destination. AI can be invaluable in helping resolve real business issues and recommend new products or services, for instance, but if they are improperly set up, they can quickly become expensive failures. It makes sense, then, to start with smaller applications of AI and then slowly expand.

IDP can be an important and key first step in that journey – an opportunity to start with automating of document processing before expanding to other functions and implementations across the enterprise. AI-powered intelligent document processing quickly demonstrates business value and instills faith in the power of AI across stakeholder groups. They will then be more willing to expand into other AI initiatives that will benefit your organization in additional ways.

Source Prolead brokers usa