Testing AI Features Without Burning Real API Credits

Every project in this series makes a real, billable API call. Running a real test suite that calls the actual API on every test run is genuinely expensive and slow — this part covers the correct, real way to test AI-powered code without either problem.

The real, core testing principle

Test the DETERMINISTIC code around the AI call thoroughly and
  directly (real, exact assertions)
Mock the AI call itself, rather than hitting the real API in tests

This directly reflects a genuine, structural fact about this series' own code: functions like part 12's find_relevant_docs() or part 19's chunk_text() are fully deterministic and testable in the normal, real way — it's specifically the actual LLM call that's expensive and non-deterministic (per the AI Fundamentals series' own sampling coverage), and that's the real, specific piece worth mocking.

Real, practical mocking with unittest.mock

# test_summarizer.py
from unittest.mock import patch, MagicMock
from summarizer import summarize

@patch("summarizer.get_client")
def test_summarize_calls_api_correctly(mock_get_client):
    mock_response = MagicMock()
    mock_response.content = [MagicMock(text="A real, mocked summary.")]
    mock_client = MagicMock()
    mock_client.messages.create.return_value = mock_response
    mock_get_client.return_value = mock_client

    result = summarize("Some real, long input text.")

    assert result == "A real, mocked summary."
    mock_client.messages.create.assert_called_once()

This real test verifies two genuinely important things without ever making a real network request: that summarize() correctly returns whatever text the API responds with, and that it actually calls the API exactly once with the expected shape — real, fast, free, and repeatable on every test run.

Testing the real, deterministic logic thoroughly, with real assertions

# test_qa_app.py
from qa_app import find_relevant_docs, KNOWLEDGE_BASE

def test_find_relevant_docs_matches_refund_question():
    results = find_relevant_docs("what if my order arrives damaged", KNOWLEDGE_BASE)
    assert any(doc["id"] == "refund-1" for doc in results)

def test_find_relevant_docs_returns_empty_for_unrelated_question():
    results = find_relevant_docs("do you offer corporate gifting", KNOWLEDGE_BASE)
    assert results == []

This is genuinely, fully testable with real, exact assertions — no mocking needed at all, since find_relevant_docs() is pure, deterministic Python logic with no AI call inside it. This directly demonstrates the real, practical value of part 12's architecture — separating retrieval (deterministic, thoroughly testable) from generation (non-deterministic, mocked) into distinct functions.

What genuinely can't be tested with a real, exact assertion

Cannot reliably assert: "the real AI response contains exactly this
  specific phrasing" — per the AI Fundamentals series' own sampling
  coverage, exact wording genuinely varies between real API calls

A real, honest test suite doesn't try to assert exact AI-generated text content — that's testing the model's own behavior, not your application's code, and it would be genuinely flaky by design. What's actually testable and worth testing: that your code calls the API correctly, handles a real error response correctly (mocking a failure, not just a success), and correctly processes whatever real response comes back.

A real, practical test for error handling

@patch("summarizer.get_client")
def test_summarize_handles_api_failure(mock_get_client):
    from anthropic import APIConnectionError
    mock_client = MagicMock()
    mock_client.messages.create.side_effect = APIConnectionError(request=MagicMock())
    mock_get_client.return_value = mock_client

    # assuming a real, graceful fallback per part 7's error handling
    result = summarize_with_error_handling("some text")
    assert "trouble" in result.lower()

This directly verifies part 7's error-handling behavior — mocking a real, specific failure (APIConnectionError) and confirming the code's genuine, graceful fallback response, rather than only ever testing the success path.

A real, practical place for occasional, genuine integration tests

A small, separate real test suite — run manually or on a real,
  scheduled basis, NOT on every commit — that DOES call the actual
  API, checking the real integration still works end to end

This is a real, honest middle ground: the vast majority of tests should mock the API for speed and cost, but a small, separate, deliberately infrequent suite of genuine integration tests catches real, actual breakage (a changed real API response format, an expired real key) that mocked tests structurally cannot detect.

Next: deploying a real Python AI application — environment configuration, secrets management, and the practical differences between local development and real production.