From 7 Habits to 12 Traits: The Evolution of the Universal Interview

 You can find them both using this one question:

Tell me about your best work (describe your biggest job challenge)?

Then dig deep 5-6 layers to uncover the attributes of excellence shown in the infographic above. When you can complete this Quality of Hire Talent Scorecard you know you've dug deep enough.

performancebasedhiring.com

The Evolution of the 12 Traits Began with Stephen Covey

Identifying these traits started in the 1980s with Stephen Covey's "7 Habits of Highly Effective People." His principle-centered approach revealed that excellence wasn't about techniques or skills – it was about character and how people approached their work. I then incorporated these habits into my interview methodology, probing for proactivity, win-win thinking, and systems perspective.

His principle-centered approach revealed that excellence wasn't about techniques or skills – it was about character and how people approached their work.

Over the years and after interviewing thousands of candidates, Covey's foundational principles manifested in specific, observable behavioral patterns that predicted success across every role and industry. This transformed interviewing from a philosophical assessment to behavioral science.

The Evolution: From Principles to Patterns

Covey taught us to look for people who "Begin with the End in Mind." This principle consistently appeared as two distinct traits in top performers: Strategic Perspective (understanding why before how) and Stakeholder Obsession (delivering outcomes, not just specifications).

His "Be Proactive" habit revealed itself through Ownership Beyond Boundaries – top performers who couldn't walk past problems even outside their job description. As important, the best demonstrated Comfort with Ambiguity, moving forward intelligently without waiting for perfect clarity.

"Think Win-Win" and "Seek First to Understand" merged into what I call Influence Without Authority. But the data showed something else: exceptional performers exhibited Communication Clarity that went beyond understanding – they translated complexity for any audience, turning Covey's empathetic listening into actionable influence.

The Universal Interview Method Reveals Skills

The question "Describe your most significant accomplishment related to [specific and realistic job challenge]" highlights what people have done with their skills, not the skills as absolutes inside a vacuum lacking context.

You can experience this yourself whenever a hiring manager demands some "Must Have" skill. In this case just ask, "How is that skill used and measured on the job?" It will be outcome based like, "Rebuild the supply chain and procurement process to be flexible enough to handle the changing tariff rules we're now experiencing."

Then ask if the hiring manager would see a person who has accomplished something comparable even if they don't have the number of years and industry experience listed. Few will object.

This is how you bridge the gap from the job description to the interview to a more accurate assessment.

More important, it's how you open the talent pool to some remarkable people who would never have been seen otherwise.

To see this process in action just dump any job description for any job into our Performance-based Hiring Talent Hub and ask it to convert it into a list of critical performance objectives. Then ask it for a list of interview questions and a scorecard. It will even post an ad for you on our private job board with a very compelling career-focused ad.

The Competitive Advantage

Organizations using this framework are able to build teams that embody Covey's vision while demonstrating measurable excellence. They hire people who don't just understand the 7 Habits intellectually but live them through these 12 observable traits.

Excellence has a pattern. Covey identified its philosophical foundation; decades of interviewing revealed its behavioral expression. The Universal Interview doesn't replace the 7 Habits – it operationalizes them, making timeless principles measurable in every hiring decision.

---

This is how you play "Moneyball for HR!" by rethinking what excellence really means and how to measure it.

The Hidden Danger of Skills-based Hiring

 I always start a new search project with the hiring manager with this question,

"What does the person in this role need to do to be successful over the course of the first year?"

This always produces 5-6 key action-oriented performance objectives like, "Build and lead a logistics team to develop a fast-reaction procurement system to handle rapidly changing international tariffs."

Then I ask, "What are the 'must have' critical skills that you won't comprise on?" This always results in 2-3 important competencies like strong communication skills or some exotic technical need.

But it's what I ask next that's the most important part of the intake meeting:

How will a person actually use (that critical skill) on the job?

In one search a hiring manager wanted an MS in Electrical Engineering for a VP Marketing role. What he actually wanted was someone to lead the development of a very complex three-year product roadmap in a rapidly changing technical environment. The person we placed was very technical but didn't have an EE degree.

In other situation the VP Finance wanted to hire a controller with a CPA and super heavy SEC reporting skills. What he actually wanted was someone who could project manage the SEC reporting function. The person we placed was no SEC expert, but she was great at managing the complexities and pressures involved in the financial reporting for a publicly-traded company.

Despite the value of converting a skill into a measurable outcome, getting the hiring manager to answer these questions was challenging. This is no longer the case with AI. We now just embed the answer into the question. For example, "For this data analyst role if seems like you'd want someone who could convert our massive product sales information into real time product line profitability analysis. Is this correct?"

---

The shift to skills-based hiring is undeniably positive. But let's be clear: it's just one step in a multi-step process. You need to be sure the skills selected are the correct ones, that they’re being evaluated properly and you can attract qualified people who are both competent AND motivated to use these skills once hired.

Unless these pieces are fully integrated all you’ll be doing is seeing more people responding to your job postings. That's not progress – that's just more noise.

And hiring competent but unmotivated people is worse than noise. It’s a bad hire.

TestGorilla is leading the way on solving some of these problems and they asked me what else could be done and to share my perspective at an upcoming event.

I simply said that it's what people DO with what they HAVE that matters, not what they HAVE, and that's why you to start every search project with this one question:

"What does this person need to do to be successful in this role?"

It turns out that every job in the world can be defined by 6-8 performance objectives that describe the actual work that needs to be done. This then leads to the next question about what are the critical skills needed to do this work. Here they are for the data analyst role.

Attracting In vs. Weeding Out Is the Real Super Skill

But here's what often gets missed: attracting people who can do this work is actually more important than verifying whether they're competent.

The best assessment tools in the world won't help if the right candidates never apply. Just as bad, is hiring people who are competent to do the work, but not motivated to do it.

However these is a solution to this dilemma. I call it “Job Branding.” To see its power just add this type of phrase in your emails or and job postings:

Use your [super skill] to accomplish [something really big and significant].

Instead of listing "15 years international procurement experience required," try:

"Your insight into worldwide distribution will help us navigate a world of changing tariffs."

For senior-staff and management roles, this approach will outperform any employer branding initiative because it speaks directly to the person’s intrinsic motivators.

The Bottom Line

Skills matter. Of course they do. But it's how someone uses those critical skills that determines success – not the skills themselves.

That's the future of skills-based hiring: attracting and hiring outstanding people who are both competent AND motivated to do the real work that needs to get done.

---

Using AI you can now convert any job into a career move and identify the critical skills in two minutes.     

The Shocking $200,000 Price Tag of Every Failed Job Posting Hire

 FYI: This story takes place in about six months. It could be yours. The moral: HR leaders need to step up and embrace AI, data analytics and and financial analysis. Thinking like a CFO will become the next HR super skill. That's "Moneyball for HR!"

---

Alice O'Hara had seen enough. After years of watching companies pour millions into HR technology and recruitment platforms, only to see employee engagement scores barely budge and turnover remain stubbornly high, she knew something had to change. As the newly appointed HR Business Partner for a mid-sized division of a global corporation, she was finally in a position to prove what she had long suspected: job postings were becoming an expensive loss leader in the talent acquisition strategy.

"We're spending enormous resources managing applications from people we'll never hire," she explained to Johan Evans, her recently hired HR analyst. "Even though about a third of our new hires come via postings, only 1-2% of those who apply actually get hired. That means we're investing significant time and money processing the other 98%. There has to be a better way."

Alice wasn't just making assumptions. Her background in Six Sigma and data analytics, combined with years of experience implementing Gallup's Q12 engagement surveys, had taught her to look beyond surface-level metrics. She had watched as Gallup's research consistently showed that only about a third of the U.S. workforce was fully engaged with their work, while 15-20% remained actively disengaged – numbers that hadn't significantly improved despite billions spent on HR technology and the recent emergence of AI.

For their first major project together, Alice tasked Johan with conducting a comprehensive study examining turnover, employee engagement and quality of hire by sourcing channel. While many HR leaders were focused on the industry's shift toward skills-based hiring over traditional credentials, Alice believed this wasn't enough to address the fundamental problems of engagement and retention.

To ensure a rigorous analysis, Alice first directed Johan to take LinkedIn Learning's "Moneyball for HR!" course, where he learned about applying predictive analytics to hiring outcomes. Armed with these concepts and some powerful Visier turnover data, Johan's findings were striking. Drawing from about 25 exit interviews with employees who had left during their first year, patterns began emerging. The data showed that employees hired through job postings consistently had higher turnover rates compared to other sourcing channels like employee referrals, search firms, boomerangs, and internal moves.

The reasons for departure were equally telling. Among those who left voluntarily, three major themes emerged: the biggest was that the role didn't match expectations representing about 60% of the responses. The rest were split between clashes with the hiring manager's style and misalignment with the company culture. These same problems didn't exist with internal moves, boomerangs or those hired via referral. In these cases the hiring manager's leadership style was the primary cause of dissatisfaction and turnover.

"What really caught my attention," Johan shared during a briefing with Alice, "was the resource imbalance. We're spending disproportionate time and money on candidates from job postings – more interviews per hire, more assessment steps, more recruiter hours, and more HR tech spend – yet our data suggests these hires are less likely to succeed."

The project caught the attention of the CFO, who offered his team's support in building a comprehensive financial model. The CFO's analysis of a typical $100,000 position revealed the stark financial reality. A successful hire delivers $135,000 in net profit their first year. In contrast, an early departure creates a $65,000 loss when factoring in reduced productivity and replacement costs. "That's a $200,000 swing for a single position," the CFO noted. "Multiply that across dozens of hires, and the impact of choosing the right sourcing channel becomes impossible to ignore."

"Multiply that across dozens of hires, and the impact of choosing the right sourcing channel becomes impossible to ignore."

As Alice reviewed the preliminary data, she couldn't help but wonder why these patterns hadn't been addressed sooner. Despite years of emphasis on big data and HR analytics, companies continued to pour resources into a recruiting channel that seemed to consistently underperform. The scale of the problem was staggering: with about 50 staff-level positions turning over in their first year, and each failed hire representing a $200,000 swing, they were looking at a $10 million real profit loss – most of it tied directly to job posting hires and the enormous expense to manage this channel

"We can't wait any longer to address this," Alice told Johan as she prepared to brief the executive team. "Every month we delay costs us another million dollars. Next month we'll be presenting the ROI for each sourcing channel to the CEO. If we're right and we do it right it will change the role of HR at our company. Forever. Let's get ready with a detailed plan of action to implement our findings."

---

Performance-based Hiring, from Lou Adler's bestseller Hire with Your Head, transforms traditional hiring by focusing on defining actual job success and evaluating candidates through their past comparable achievements. A top labor attorney considers this the benchmark for hiring stronger and more diverse talent. The company offers a series of live and online training programs for recruiters and hiring managers who want to achieve more Win-Win Hiring outcomes.

Be sure to join our "Moneyball for HR!" club for some other great hiring ideas or contact us right away if you can't wait.

Loyalty in Your Customer's Hand. Why the Best Programs Stopped Counting Points

 

I met the founders of Sweet Tooth way back in 2012 at the first Big Dam Run in Vegas. You know them now as Smile.io, but back then they were Sweet Tooth Rewards, a scrappy loyalty app built on Magento. Imagine a bunch of Canadians coming to the Nevada desert in the winter to run a half marathon. If that isn't loyalty, I don't know what is.

That was over a decade ago. Smile.io was just getting started, and loyalty in eCommerce meant punch cards and "buy 10 get 1 free" coupons. Simple. Transactional. Easy to understand.

Fast forward to 2026 and the loyalty landscape looks nothing like it did at that starting line in Vegas.

More Programs, Less Loyalty

The global loyalty management market is worth over $8.6 billion and projected to exceed $18.2 billion by the end of this year (MarketsandMarkets, 2022). Loyalty program enrollment is at all-time highs. Every brand, from the corner coffee shop to the Fortune 500, has some kind of rewards program.

And yet true loyalty, the deep trust-based connection between a customer and a brand, fell to 29% in 2025. Down from 31% the year before (SAP Emarsys Customer Loyalty Index, 2025).

More money on loyalty. More signups. Less actual loyalty.

Something is broken.

The problem isn't that loyalty programs don't work. The data says they do. 90% of loyalty program owners report positive ROI, with average returns of 4.8x (Antavo Global Customer Loyalty Report, 2024). Members who redeem rewards spend 3.1x more than those who don't (Antavo, 2024). A 5% increase in retention can drive a 25-95% increase in profit (Bain & Company).

The problem is that most programs are still built for 2012. Points for purchases. Generic discounts. One-size-fits-all tiers. The same playbook everyone else is running.

When every brand offers the same loyalty experience, none of them feel special. That's loyalty fatigue. And it's real.

The Old Model Is Table Stakes

Let me be blunt. If your loyalty program is still just "spend money, earn points, get a discount," you're not running a loyalty program. You're running a rebate system.

Customers expect that now. It's table stakes. It's the minimum. And the minimum doesn't create emotional connection.

Think about your own behavior. How many loyalty programs are you a member of right now? Ten? Twenty? How many of them actually change your purchasing decisions? Probably two or three at most.

The ones that matter to you are the ones that go beyond the transaction. The ones that make you feel like you belong to something. The ones that know you.

Nearly 7 in 10 consumers modify how much they spend to maximize loyalty benefits (Bond Brand Loyalty). But only when the program gives them a reason to care.

What the Best Programs Look Like Now

The platforms driving loyalty in 2026 look fundamentally different from what we had a decade ago.

AI-driven personalization is no longer optional. 51.4% of loyalty programs now use AI, up from 37.1% just a year ago (Antavo GCLR, 2025/2026). And these aren't gimmicky chatbots. We're talking about propensity modeling that calculates the precise likelihood of specific customer actions in real time. "Customer A is 85% likely to churn in the next 30 days." That kind of precision.

The shift is from grouping customers by demographics to understanding them as individuals. Their behavior, context, location, even the weather.

Mobile wallets are becoming loyalty hubs. Starbucks proved this with over 34 million active rewards members on their wallet-based app (Starbucks Investor Relations, 2026). Now platforms like Rivo are bringing Apple Wallet and Google Wallet loyalty card integration to Shopify brands. Your loyalty card lives right next to your credit card. No app to download. No login to remember.

Experiential rewards are overtaking discounts. What's more memorable, a $5 discount or an exclusive workshop with the brand's founder? Nike runs "Member Days" and exclusive product drops tied to engagement history. Sephora built an entire online beauty community around their Beauty Insider program. One global fashion retailer reported a 33% boost in high-tier retention by rewarding customers for attending runway livestreams, sharing outfits on Instagram, and completing sustainability pledges (Kognitiv/Antavo).

Zero-party data is the new currency. With third-party cookies dying and privacy regulations tightening, loyalty programs have become one of the most effective tools for collecting data customers willingly share. 90.7% of loyalty program owners now use loyalty data in pricing and promotions (Antavo GCLR, 2026). But the smart ones are going further, feeding that data into merchandising, product development, and customer service.

The Platform Landscape in 2026

If you're evaluating loyalty platforms right now, the market has matured significantly.

Smile.io is still the easiest on-ramp. Over 100,000 businesses use it (Smile.io). You can launch a template-based program in under an hour. Points, referrals, VIP tiers. Free plan for up to 200 monthly orders. Those Canadians who ran a half marathon in the desert built something that lowers the barrier for everyone.

If you're all-in on Shopify, look at Rivo. Built exclusively for Shopify and Shopify Plus, serving 9,000+ brands including HexClad Cookware Ridge, and Dr. Squatch (Rivo). Their claim to fame is performance. Theme app extensions that load under 100ms. HexClad generated $450,000 in referral revenue in their first 90 days, a 92x ROI (Rivo case study). They're 100% bootstrapped, zero VC, which tells you something about the business model.

Yotpo's play is bundling. Reviews, UGC, SMS marketing, and loyalty in a single platform. A customer gets points for uploading a photo review, and that photo feeds into your site gallery. That cross-module connection is hard to replicate with standalone tools.

LoyaltyLion takes the opposite approach. 100% focused on loyalty. No reviews, no SMS, just a deep, data-driven loyalty engine. They support Shopify, BigCommerce and Adobe Commerce which makes them versatile across platforms.

At the enterprise level, Antavo AI Loyalty Cloud and Open Loyalty are worth evaluating. Antavo's "Timi AI" brings agentic AI to loyalty management. No-code builders for earning logic, tiers, reward catalogs, and campaign workflows. Omnichannel support including in-store, loyalty kiosks, and physical card solutions. Multi-country, multi-currency, multi-brand coalition programs. Their client list includes KFC, SKIMS, and Benefit Cosmetics. Open Loyalty takes an API-first approach and publishes the widely-cited annual Loyalty Program Trends report.

For Magento and Adobe Commerce merchants specifically, your options have expanded. Antavo, LoyaltyLion, Zinrelo, Open Loyalty, and bLoyal all support Adobe Commerce. Zinrelo is worth a look for B2B loyalty with predictive churn modeling and AI-driven segmentation. They also integrate with VTEX, Salesforce Commerce Cloud, and Oracle Netsuite.

A note on the broader market. G2's 2026 rankings for loyalty management software include Salesforce Marketing Cloud, SAP Emarsys, Impact.com, LoyaltyLion, Smile.io, Kangaroo Rewards, and BON Loyalty among their top-rated platforms. Some of the names on G2's list (Xoxoday, AdvantageClub.ai, Snappy) are more focused on employee recognition than eCommerce loyalty, so know what you're looking at when you browse those lists. BON Loyalty is worth a look if you want a user-friendly interface with strong rewards customization. Kangaroo Rewards brings solid omnichannel capabilities. And SAP Emarsys offers advanced data-driven marketing tied to loyalty, which makes sense if you're already in the SAP ecosystem.

Before you compare feature lists, ask one question: does it talk to Klaviyo? Does it talk to your POS? If your loyalty platform can't see what your email platform sees, you're going to spend the next year duct-taping data together. I've watched brands do this. It's painful.

Loyalty as an Engagement Hub

The best loyalty programs in 2026 aren't just rewarding purchases. They're rewarding engagement.

Writing reviews. Sharing on social media. Referring friends. Completing challenges. Attending events. Volunteering. Living the brand's values.

This isn't a nice-to-have anymore. Emotionally connected customers demonstrate 306% higher lifetime value (Motista, 2018). Read that number again. Three hundred and six percent.

The gamification market is expected to reach $48.72 billion by 2029 (Mordor Intelligence), and the reason is simple. Tiers, challenges, and status levels drive behavior. When customers see a path to a higher level, they engage more.

But the brands doing this best aren't just gamifying for the sake of it. They're building community. They're creating belonging. They're making their loyalty program the front door to a relationship, not just a transaction tracker.

71% of British adults expect loyalty programs to promote sustainable causes (Mando-Connect/YouGov). Brands like Costa Coffee, Madewell, and REI are embedding sustainability rewards. Points for using reusable cups. Credit for recycling old products. Recognition for choosing lower-impact shipping.

KEEN Corps. The Example Worth Studying

This brings me to KEEN, the outdoor footwear brand out of Portland. In September 2021, they launched KEEN Corps, described as the world's first volunteerism-based loyalty program (KEEN/Points of Light). The name was inspired by the Civilian Conservation Corps of the 1930s.

In KEEN Corps, volunteering one hour earns you 25 points. Donating one dollar earns you 25 points. Spending ten dollars earns you 25 points.

They made community impact equal to revenue. The math actually works that way.

Their tier names tell you everything about the brand's personality. KEEN. KEENer. KEENest. The top tier gets you onto an exclusive product tester team. Not just discounts. Access.

The early results were strong. Within two months of launch, 11,000+ members with 5% weekly growth. Over 6,600 volunteer hours logged. $75,000 in KEEN Effect grants pledged to nonprofits (KEEN/Points of Light, 2021). 100% of donated dollars go to organizations making the outdoors and trades more accessible.

KEEN is a family-owned, values-led brand. They're not giving points for purchases and calling it a day. They're using their loyalty program as a community organizing tool. And their customers are showing up. Literally showing up. To volunteer.

That's not a rebate system. That's loyalty.

What This Means for Your Brand

If you don't have a loyalty program yet, just pick one and launch it this week. Seriously. A Smile.io free plan takes 45 minutes to set up. You'll learn more from running a basic program for 90 days than from six months of evaluating platforms.

If you already have a program, ask yourself when you last logged in and looked at the data. Most brands set up their loyalty program and forget about it. The points accumulate, the discounts go out, and nobody is paying attention to what customers are actually doing with the program.

The brands winning at loyalty right now are the ones treating their program like a product, not a feature. They're A/B testing reward structures. They're feeding loyalty data into merchandising decisions. They're asking customers what they want and then building it.

And they're thinking about what they actually stand for. KEEN rewards volunteering because they genuinely care about the outdoors. That's not a marketing play. That's who they are. Your customers can tell the difference.

The hard part is not the technology. Every platform I listed above can do the technical work. The hard part is deciding what your brand actually stands for, and then building a program around that instead of copying what everyone else is doing.

Join Us at eTail

I'm at eTail™ Palm Springs this week, and the timing of this article is intentional. Today at 2:40 PM PST, Sam Buckingham from KEEN is doing a fireside chat on Track 1. "Loyalty in the Palm of Their Hand: Maximizing Engagement Through Apps, Cards, and Beyond." He's going to dig into building loyalty triggers at different points of the customer journey and allowing customers to access programs in different ways.

If you're here at the conference, come join us. If you're not, I'll share the key takeaways in a future post.

The brands that figure out loyalty in this next era won't be the ones with the biggest points multipliers. They'll be the ones that give their customers something worth being loyal to.

---

Disclaimer: I'm not endorsing any specific loyalty platform mentioned in this article. I have no financial relationship with any of them. This is simply a look at where the loyalty space is heading and the platforms I'm seeing in the market.

Sources

• Antavo Global Customer Loyalty Report 2024 (https://antavo.com/reports/global-customer-loyalty-report-2024/)
• Antavo Global Customer Loyalty Report 2025/2026 (https://antavo.com/blog/ai-transformation/)
• Bain & Company, Frederick Reichheld, "Zero Defections: Quality Comes to Services," HBR 1990 (https://www.bain.com/insights/retaining-customers-is-the-real-challenge/)
• Bond Brand Loyalty Report (https://bondbl.com/bond-loyalty-report/)
• KEEN Footwear / Points of Light Press Release, November 2021 (https://www.prnewswire.com/news-releases/keen-footwear-and-points-of-light-join-to-fight-global-stress-and-uncertainty-launching-largest-volunteer-resource-ahead-of-giving-tuesday-301426133.html)
• KEEN Footwear Press Release, September 2021 (https://www.prnewswire.com/news-releases/keen-footwear-taps-into-global-desire-for-change-launches-keen-corps-loyalty-program-rewarding-good-deeds-301378072.html)
• Kognitiv, "Top 12 Best Loyalty Program Software Solutions" (https://www.kognitiv.com/articles-post/top-12-best-loyalty-program-software-solutions-in-the-world)
• Mando-Connect / YouGov, "What the British Want from Loyalty Programmes 3.0" (https://thewisemarketer.com/new-research-from-the-uk-what-the-british-want-from-loyalty-programmes-3-0/)
• MarketsandMarkets, Loyalty Management Market Report 2022 (https://www.globenewswire.com/news-release/2022/01/25/2372142/0/en/)
• Mordor Intelligence, Gamification Market Size and Forecast 2024-2029 (https://www.mordorintelligence.com/industry-reports/gamification-market/market-size)
• Motista, "Leveraging the Value of Emotional Connection for Retailers," 2018 (https://www.prnewswire.com/news-releases/new-retail-study-shows-marketers-under-leverage-emotional-connection-300720049.html)
• Rivo HexClad Case Study (https://www.rivo.io/case-studies/hexclad)
• SAP Emarsys Customer Loyalty Index 2025 (https://emarsys.com/learn/white-papers/customer-loyalty-index-2025-global/)
• Starbucks Investor Relations (https://investor.starbucks.com/)

The Search for the Sweet Spot in a Linear Regression with Numeric Features

 

Consistent with the principle of Occam’s razor, starting simple often leads to the most profound insights, especially when piecing together a predictive model. In this post, using the Ames Housing Dataset, we will first pinpoint the key features that shine on their own. Then, step by step, we’ll layer these insights, observing how their combined effect enhances our ability to forecast accurately. As we delve deeper, we will harness the power of the Sequential Feature Selector (SFS) to sift through the complexities and highlight the optimal combination of features. This methodical approach will guide us to the “sweet spot” — a harmonious blend where the selected features maximize our model’s predictive precision without overburdening it with unnecessary data.

Kick-start your project with my book Next-Level Data Science. It provides self-study tutorials with working code.

Let’s get started.

The Search for the Sweet Spot in a Linear Regression with Numeric Features
Photo by Joanna Kosinska. Some rights reserved.

Overview

This post is divided into three parts; they are:

• From Single Features to Collective Impact
• Diving Deeper with SFS: The Power of Combination
• Finding the Predictive “Sweet Spot”

From Individual Strengths to Collective Impact

Our first step is to identify which features out of the myriad available in the Ames dataset stand out as powerful predictors on their own. We turn to simple linear regression models, each dedicated to one of the top standalone features identified based on their predictive power for housing prices.

# Load the essential libraries and Ames dataset from sklearn.model_selection import cross_val_score from sklearn.linear_model import LinearRegression import pandas as pd   Ames = pd.read_csv("Ames.csv").select_dtypes(include=["int64", "float64"]) Ames.dropna(axis=1, inplace=True) X = Ames.drop("SalePrice", axis=1) y = Ames["SalePrice"]   # Initialize the Linear Regression model model = LinearRegression()   # Prepare to collect feature scores feature_scores = {}   # Evaluate each feature with cross-validation for feature in X.columns:     X_single = X[[feature]]     cv_scores = cross_val_score(model, X_single, y)     feature_scores[feature] = cv_scores.mean()   # Identify the top 5 features based on mean CV R² scores sorted_features = sorted(feature_scores.items(), key=lambda item: item[1], reverse=True) top_5 = sorted_features[0:5]   # Display the top 5 features and their individual performance for feature, score in top_5:     print(f"Feature: {feature}, Mean CV R²: {score:.4f}")

This will output the top 5 features that can be used individually in a simple linear regression:

Feature: OverallQual, Mean CV R²: 0.6183 Feature: GrLivArea, Mean CV R²: 0.5127 Feature: 1stFlrSF, Mean CV R²: 0.3957 Feature: YearBuilt, Mean CV R²: 0.2852 Feature: FullBath, Mean CV R²: 0.2790

Curiosity leads us further: what if we combine these top features into a single multiple linear regression model? Will their collective power surpass their individual contributions?

# Extracting the top 5 features for our multiple linear regression top_features = [feature for feature, score in top_5]   # Building the model with the top 5 features X_top = Ames[top_features]   # Evaluating the model with cross-validation cv_scores_mlr = cross_val_score(model, X_top, y, cv=5, scoring="r2") mean_mlr_score = cv_scores_mlr.mean()   print(f"Mean CV R² Score for Multiple Linear Regression Model: {mean_mlr_score:.4f}")

The initial findings are promising; each feature indeed has its strengths. However, when combined in a multiple regression model, we observe a “decent” improvement—a testament to the complexity of housing price predictions.

Mean CV R² Score for Multiple Linear Regression Model: 0.8003

This result hints at untapped potential: Could there be a more strategic way to select and combine features for even greater predictive accuracy?

Diving Deeper with SFS: The Power of Combination

As we expand our use of Sequential Feature Selector (SFS) from 𝑛 =1 to 𝑛 =5, an important concept comes into play: the power of combination. Let’s illustrate as we build on the code above:

# Perform Sequential Feature Selector with n=5 and build on above code from sklearn.feature_selection import SequentialFeatureSelector   sfs = SequentialFeatureSelector(model, n_features_to_select=5) sfs.fit(X, y)   selected_features = X.columns[sfs.get_support()].to_list() print(f"Features selected by SFS: {selected_features}")   scores = cross_val_score(model, Ames[selected_features], y) print(f"Mean CV R² Score using SFS with n=5: {scores.mean():.4f}")

Choosing 𝑛 =5 doesn’t merely mean selecting the five best standalone features. Rather, it’s about identifying the set of five features that, when used together, optimize the model’s predictive ability:

Features selected by SFS: ['GrLivArea', 'OverallQual', 'YearBuilt', '1stFlrSF', 'KitchenAbvGr'] Mean CV R² Score using SFS with n=5: 0.8056

This outcome is particularly enlightening when we compare it to the top five features selected based on their standalone predictive power. The attribute “FullBath” (not selected by SFS) was replaced by “KitchenAbvGr” in the SFS selection. This divergence highlights a fundamental principle of feature selection: it’s the combination that counts. SFS doesn’t just look for strong individual predictors; it seeks out features that work best in concert. This might mean selecting a feature that, on its own, wouldn’t top the list but, when combined with others, improves the model’s accuracy.

If you wonder why this is the case, the features selected in the combination should be complementary to each other rather than correlated. In this way, each new feature provides new information for the predictor instead of agreeing with what is already known.

Finding the Predictive “Sweet Spot”

The journey to optimal feature selection begins by pushing our model to its limits. By initially considering the maximum possible features, we gain a comprehensive view of how model performance evolves by adding each feature. This visualization serves as our starting point, highlighting the diminishing returns on model predictability and guiding us toward finding the “sweet spot.” Let’s start by running a Sequential Feature Selector (SFS) across the entire feature set, plotting the performance to visualize the impact of each addition:

# Performance of SFS from 1 feature to maximum, building on code above: import matplotlib.pyplot as plt   # Prepare to store the mean CV R² scores for each number of features mean_scores = []   # Iterate over a range from 1 feature to the maximum number of features available for n_features_to_select in range(1, len(X.columns)):     sfs = SequentialFeatureSelector(model, n_features_to_select=n_features_to_select)     sfs.fit(X, y)     selected_features = X.columns[sfs.get_support()]     score = cross_val_score(model, X[selected_features], y, cv=5, scoring="r2").mean()     mean_scores.append(score)   # Plot the mean CV R² scores against the number of features selected plt.figure(figsize=(10, 6)) plt.plot(range(1, len(X.columns)), mean_scores, marker="o") plt.title("Performance vs. Number of Features Selected") plt.xlabel("Number of Features") plt.ylabel("Mean CV R² Score") plt.grid(True) plt.show()

The plot below demonstrates how model performance improves as more features are added but eventually plateaus, indicating a point of diminishing returns:

Comparing the effect of adding features to the predictor

From this plot, you can see that using more than ten features has little benefit. Using three or fewer features, however, is suboptimal. You can use the “elbow method” to find where this curve bends and determine the optimal number of features. This is a subjective decision. This plot suggests anywhere from 5 to 9 looks right.

Armed with the insights from our initial exploration, we apply a tolerance (tol=0.005) to our feature selection process. This can help us determine the optimal number of features objectively and robustly:

# Apply Sequential Feature Selector with tolerance = 0.005, building on code above sfs_tol = SequentialFeatureSelector(model, n_features_to_select="auto", tol=0.005) sfs_tol.fit(X, y)   # Get the number of features selected with tolerance n_features_selected = sum(sfs_tol.get_support())   # Prepare to store the mean CV R² scores for each number of features mean_scores_tol = []   # Iterate over a range from 1 feature to the Sweet Spot for n_features_to_select in range(1, n_features_selected + 1):     sfs = SequentialFeatureSelector(model, n_features_to_select=n_features_to_select)     sfs.fit(X, y)     selected_features = X.columns[sfs.get_support()]     score = cross_val_score(model, X[selected_features], y, cv=5, scoring="r2").mean()     mean_scores_tol.append(score)   # Plot the mean CV R² scores against the number of features selected plt.figure(figsize=(10, 6)) plt.plot(range(1, n_features_selected + 1), mean_scores_tol, marker="o") plt.title("The Sweet Spot: Performance vs. Number of Features Selected") plt.xlabel("Number of Features") plt.ylabel("Mean CV R² Score") plt.grid(True) plt.show()

This strategic move allows us to concentrate on those features that provide the highest predictability, culminating in the selection of 8 optimal features:

Finding the optimal number of features from a plot

We can now conclude our findings by showing the features selected by SFS:

# Print the selected features and their performance, building on the above: selected_features = X.columns[sfs_tol.get_support()] print(f"Number of features selected: {n_features_selected}") print(f"Selected features: {selected_features.tolist()}") print(f"Mean CV R² Score using SFS with tol=0.005: {mean_scores_tol[-1]:.4f}")

Number of features selected: 8 Selected features: ['GrLivArea', 'LotArea', 'OverallQual', 'OverallCond', 'YearBuilt', '1stFlrSF', 'BedroomAbvGr', 'KitchenAbvGr'] Mean CV R² Score using SFS with tol=0.005: 0.8239

By focusing on these 8 features, we achieve a model that balances complexity with high predictability, showcasing the effectiveness of a measured approach to feature selection.

Further Reading

APIs

• sklearn.feature_selection.SequentialFeatureSelector API

Tutorials

• Sequential Feature Selection by Sebastian Raschka

Ames Housing Dataset & Data Dictionary

• Ames Dataset
• Ames Data Dictionary

Summary

Through this three-part post, you have embarked on a journey from assessing the predictive power of individual features to harnessing their combined strength in a refined model. Our exploration has demonstrated that while more features can enhance a model’s ability to capture complex patterns, there comes a point where additional features no longer contribute to improved predictions. By applying a tolerance level to the Sequential Feature Selector, you have honed in on an optimal set of features that propel our model’s performance to its peak without overcomplicating the predictive landscape. This sweet spot—identified as eight key features—epitomizes the strategic melding of simplicity and sophistication in predictive modeling.

Specifically, you learned:

• The Art of Starting Simple: Beginning with simple linear regression models to understand each feature’s standalone predictive value sets the foundation for more complex analyses.
• Synergy in Selection: The transition to the Sequential Feature Selector underscores the importance of not just individual feature strengths but their synergistic impact when combined effectively.
• Maximizing Model Efficacy: The quest for the predictive sweet spot through SFS with a set tolerance teaches us the value of precision in feature selection, achieving the most with the least.

Do you have any questions? Please ask your questions in the comments below, and I will do my best to answer.