Machine Learning Framing

Overview

Machine Learning Framing is a systematic methodology for translating business problems into well-defined machine learning projects with clear evaluation criteria. Based on the AWS Well-Architected Machine Learning Lens¹, it bridges the critical gap between business ambition and technical implementation, ensuring ML initiatives deliver measurable value rather than becoming expensive experiments.

Core Principles

1. Business-First Thinking

• Start with business problems, not ML solutions
• Quantify impact before building
• Define success in business terms
• Maintain stakeholder alignment

2. Problem-Solution Fit

• Match ML approach to problem type
• Validate technical feasibility early
• Consider simpler alternatives first
• Build only what’s necessary

3. Measurable Outcomes

• Establish evaluation criteria upfront
• Connect technical metrics to business value
• Create feedback loops
• Enable objective go/no-go decisions

4. Iterative Refinement

• Start with hypotheses, not certainties
• Test assumptions quickly
• Adapt based on learnings
• Fail fast and cheap

5. Cross-Functional Alignment

• Unite business and technical teams
• Share common language
• Agree on success metrics
• Maintain continuous communication

The ML Framing Process

Phase 1: Business Understanding

Objective

Identify and prioritize the business problem, quantify its impact, and define success in measurable business terms.

Key Activities

1. Problem Discovery ``` Questions to Answer:
   • What specific business challenge exists?
   • Who experiences this problem?
   • How often does it occur?
   • What’s the current cost/impact? ```
2. Impact Quantification ``` Metrics to Calculate:
   • Revenue impact
   • Cost implications
   • Time savings potential
   • Customer satisfaction effect
   • Competitive advantage ```
3. Success Vision ``` Define Clear Outcomes:
   • “We will know we’ve succeeded when…”
   • Specific business metrics that will improve
   • Target values for each metric
   • Timeline for achievement ```

Deliverables

• Business problem statement
• Stakeholder map
• Impact analysis
• Success criteria document

Phase 2: Technical Problem Framing

Objective

Transform the business problem into a concrete ML problem with defined inputs, outputs, and constraints.

Key Activities

1. ML Problem Classification

   Common ML Problem Types:
      
   Classification:
   - Binary: Yes/No, Pass/Fail, Fraud/Legitimate
   - Multi-class: Category A/B/C/D
   - Multi-label: Multiple tags per item
      
   Regression:
   - Continuous value prediction
   - Forecasting
   - Scoring
      
   Other Types:
   - Clustering (grouping similar items)
   - Recommendation (suggesting relevant items)
   - Anomaly Detection (finding outliers)
   - Time Series (temporal predictions)
   

2. Input-Output Mapping ``` Define Precisely:
   • Inputs: What data will the model observe?
   • Outputs: What should the model predict?
   • Constraints: Latency, accuracy, regulatory requirements
   • Data Requirements: Volume, quality, availability ```
3. Technical Feasibility Assessment ``` Evaluate:
   • Data availability and quality
   • Technical complexity
   • Infrastructure requirements
   • Skill requirements
   • Timeline constraints ```

Deliverables

• ML problem type selection
• Input-output specification
• Constraint documentation
• Feasibility assessment

Phase 3: Evaluation Criteria Definition

Objective

Establish clear, measurable criteria for evaluating ML solution success at both technical and business levels.

Key Activities

1. Technical Metrics Selection ``` For Classification:
   • Accuracy: Overall correctness
   • Precision: Positive prediction accuracy
   • Recall: Positive case coverage
   • F1 Score: Balanced measure
   • AUC-ROC: Threshold-independent performance

   For Regression:

   • RMSE: Root Mean Square Error
   • MAE: Mean Absolute Error
   • R²: Variance explained
   • MAPE: Percentage error ```
2. Business Metrics Mapping ``` Connect Technical to Business:
   • If precision increases by X%, revenue increases by $Y
   • If recall improves by X%, we save Z hours/week
   • If RMSE decreases by X, customer satisfaction increases by Y points ```
3. Threshold Setting ``` Define Three Levels:
   • Minimum Viable: Below this, project fails
   • Target: Expected performance level
   • Stretch: Exceptional performance ```

Deliverables

• Technical metric selection with thresholds
• Business metric mapping
• Evaluation methodology
• Success criteria matrix

Implementation Framework

Step 1: Discovery Workshop (4-6 hours)

Participants

• Business stakeholders
• Technical team
• End users
• Data owners

Agenda

1. Business problem deep-dive (90 min)
2. Current process mapping (60 min)
3. Data availability review (60 min)
4. Initial ML approach brainstorming (60 min)
5. Success criteria discussion (60 min)

Step 2: Technical Translation (2-3 hours)

Participants

• ML practitioners
• Data engineers
• Solution architects

Focus Areas

• Convert business problem to ML problem
• Assess technical feasibility
• Identify data requirements
• Propose evaluation approach

Step 3: Alignment Review (2 hours)

Participants

• All stakeholders

Objectives

• Validate problem-solution fit
• Agree on success metrics
• Establish evaluation plan
• Make go/no-go decision

Step 4: Documentation

ML Framing Record Contents

1. Executive Summary (1 page)
2. Business Understanding (1-2 pages)
3. Technical Problem Framing (1-2 pages)
4. Evaluation Criteria (1 page)
5. Risk Assessment (1 page)
6. Implementation Plan (1 page)

Common Pitfalls and Mitigations

Pitfall 1: Solution-First Mentality

Problem: “We need deep learning for customer service”

Mitigation: Always start with the business problem, not the technology. Ask “What customer service problem are we solving?” before choosing any ML approach.

Pitfall 2: Vague Success Criteria

Problem: “The model should be accurate”

Mitigation: Define specific, measurable thresholds: “The model should achieve 92% precision with 88% recall on the test set, translating to 30% reduction in manual review time.”

Pitfall 3: Ignoring Business Constraints

Problem: “Our model achieves 99.9% accuracy but takes 5 seconds per prediction”

Mitigation: Include all constraints in initial framing: “Must achieve >90% accuracy with <100ms latency to meet user experience requirements.”

Pitfall 4: Data Quality Assumptions

Problem: “We’ll have perfect data by the time we need it”

Mitigation: Assess data quality early and plan for reality: “Current data has 15% missing values; our approach must handle this gracefully.”

Pitfall 5: Analysis Paralysis on Metrics

Problem: “We can’t start until we know the exact right thresholds for every metric”

Mitigation: Start with informed estimates, measure everything, gather feedback for context, then refine based on real-world learning. Perfect metrics emerge through iteration, not planning.

Pitfall 6: Treating Metrics as Fixed Requirements

Problem: “We defined 90% accuracy in our framing, so that’s what we must achieve”

Mitigation: View initial metrics as hypotheses to test, not contracts to fulfill. Real-world feedback often reveals that different metrics matter more than originally thought.

Evaluation Methods

Offline Evaluation

• Historical data testing
• Cross-validation
• Hold-out test sets
• Error analysis by segment

Online Evaluation

• A/B testing framework
• Canary deployments
• Gradual rollouts
• User feedback collection

Business Impact Measurement

• Pre/post metric comparison
• Control group analysis
• ROI calculation
• Stakeholder satisfaction

Continuous Monitoring

• Model performance tracking
• Data drift detection
• Business metric correlation
• Feedback loop implementation

The Metric Evolution Lifecycle

Embrace Metric Uncertainty

A critical insight often missing from ML frameworks: You won’t know the right metrics or thresholds initially, and that’s expected. The ML Framing Record should be a living document that evolves as you learn.

Many organizations get paralyzed trying to define perfect metrics upfront. They delay projects for months debating whether they need 85% or 92% accuracy, or if response time should be 100ms or 200ms. This perfectionist approach kills more ML projects than technical challenges ever do.

The truth is: Your first metrics are educated guesses. They’re starting points for learning, not final destinations. The real value comes from measuring, learning, and adapting based on actual user feedback and business impact.

The Iterative Refinement Process

Phase 1: Initial Hypothesis (Pre-Development)

• Set metrics based on business needs and industry benchmarks
• Document assumptions explicitly
• Mark thresholds as “initial estimates”
• Plan for measurement infrastructure
• Key Mindset: “These are our best guesses based on current knowledge”

Phase 2: Reality Check (Early Development)

• Measure actual performance against initial targets
• Identify gaps between expectations and capabilities
• Collect early user feedback on prototypes
• Begin understanding metric interactions
• Key Learning: “Our assumptions about what matters are being tested”

Phase 3: Contextual Learning (Pilot/Beta)

• Gather qualitative feedback to contextualize numbers
• Understand why certain metrics matter more than others
• Discover unexpected important metrics
• Learn threshold sensitivities
• Key Insight: “User satisfaction often correlates with different metrics than we expected”

Phase 4: Metric Evolution (Production)

• Refine thresholds based on real impact
• Add segmented metrics for different use cases
• Deprecate metrics that don’t drive value
• Introduce new metrics based on learnings
• Key Wisdom: “The metrics that matter most emerge from real-world use”

Example Evolution Pattern

Month 0 (Framing):

- Accuracy: >90% (industry standard)
- Latency: <2 seconds (seems reasonable)
- Coverage: Not yet identified as important

Month 2 (Pilot):

- Accuracy: 85% achieved, users satisfied
- Latency: 3 seconds, acceptable with high accuracy
- Coverage: Discovered as critical - users care more about 
  completeness than precision

Month 6 (Mature):

- Coverage: >95% (primary metric)
- Accuracy: >85% for routine, >95% for financial
- Latency: <5 seconds acceptable
- Confidence Scoring: New metric for trust
- Segment-specific thresholds established

Feedback Integration Framework

1. Quantitative + Qualitative: Numbers tell what, feedback tells why
2. Stakeholder Prioritization: Different users value different metrics
3. Business Context: Connect metric changes to business outcomes
4. Continuous Adjustment: Monthly metric review cycles
5. Documentation: Track why metrics evolved for future projects

The Learning Mindset for Metrics

Traditional Approach (Often Fails):

• Define metrics → Build to spec → Measure success/failure
• Treats metrics as fixed requirements
• Success = hitting predefined numbers
• Failure = missing the targets

Iterative Approach (Recommended):

• Hypothesize metrics → Build to learn → Measure and adapt
• Treats metrics as learning tools
• Success = understanding what actually matters
• “Failure” = valuable learning opportunity

Practical Example: E-commerce Search

Initial Metrics (Month 0):

• Search relevance: 95% precision required
• Response time: <500ms
• Click-through rate: >40%

After User Testing (Month 2):

• Users prefer recall over precision (“Show me everything related”)
• Will wait 2 seconds for comprehensive results
• Click-through less important than “search refinement” behavior

Evolved Metrics (Month 6):

• Search coverage: >90% (primary)
• Result diversity score (new metric)
• Query refinement rate: <20% (indicates good first results)
• Response time: <2s for complex, <500ms for simple
• Precision: 75% acceptable with good ranking

When to Use ML Framing

Ideal Scenarios

1. Complex Pattern Recognition
   • Too many variables for rules
   • Patterns change over time
   • Human expertise hard to codify
2. Scale Challenges
   • Volume exceeds human capacity
   • Speed requirements beyond manual
   • Consistency needs across scale
3. Optimization Opportunities
   • Multiple objectives to balance
   • Large solution space
   • Dynamic environments

When to Avoid ML

1. Simple Rule-Based Solutions Work
   • Clear if-then logic
   • Limited variables
   • Stable patterns
2. Insufficient Data
   • Low volume or quality
   • No historical patterns
   • Privacy restrictions
3. Unjustifiable Complexity
   • High maintenance burden
   • Limited team expertise
   • Poor cost-benefit ratio

Tools and Templates

ML Framing Record Template

Comprehensive worksheet covering:

• Business understanding
• Technical problem framing
• Evaluation criteria
• Risk assessment
• Go/no-go decisions

Stakeholder Alignment Canvas

Visual tool for mapping:

• Stakeholder interests
• Success definitions
• Concern areas
• Communication needs

Metric Mapping Matrix

Framework connecting:

• Technical metrics
• Business outcomes
• Threshold values
• Measurement methods

Measuring Success

Leading Indicators

• Time from idea to framed problem
• Stakeholder alignment score
• Data readiness assessment
• Technical feasibility confidence

Lagging Indicators

• Projects meeting success criteria
• ROI achievement rate
• Time to production
• Business metric improvement

Continuous Improvement

• Post-project reviews
• Framework refinements
• Best practice documentation
• Knowledge sharing sessions

Real-World Applications

E-commerce Personalization

Business Problem: 68% cart abandonment rate

ML Framing: Multi-class classification to predict next likely purchase based on browsing history, with success measured by 15% reduction in abandonment rate.

Result: Clear problem definition led to focused solution achieving 22% reduction.

Manufacturing Quality Control

Business Problem: 3% defect rate costing $800K annually

ML Framing: Binary image classification with 95% recall requirement to catch defects, success measured by reducing escape rate below 0.5%.

Result: Precise framing enabled targeted model development meeting all criteria.

Customer Service Routing

Business Problem: 48-hour average response time

ML Framing: Multi-class text classification to route inquiries, with success defined as 75% reduction in response time while maintaining satisfaction scores.

Result: Well-framed problem led to phased implementation exceeding targets.

Key Takeaways

1. Frame First, Build Second: A well-framed problem is half-solved
2. Business Drives Technical: Start with business value, not ML capabilities
3. Measure What Matters: Define success before starting development
4. Iterate and Learn: Use framing as living document, not one-time exercise
5. Align Stakeholders: Shared understanding prevents expensive pivots
6. Embrace Metric Evolution: Start with best guesses, refine through real-world feedback
7. Context Over Precision: User satisfaction trumps perfect technical metrics
8. Progress Over Perfection: Launching with “good enough” metrics beats endless planning
9. Learning Over Failing: Missing initial targets provides valuable insights
10. Adaptation Over Adherence: Be willing to completely change your success definition

Related Frameworks

• Design Thinking - For human-centered problem definition
• Lean Startup Methodology - For iterative development
• Agile Transformation - For implementation approach
• Data Strategy Framework - For data readiness

References