Skip to content

Live Contract Gate

Category: Gate Pattern

Overview

A live contract gate asserts that what the code believes about an external system is true of that system. It is the one gate a mocked suite cannot be: a suite built on doubles verifies the code against its own assumptions, and a wrong assumption is invisible from the inside. The gate runs against a real instance, is skipped unless one is configured, and asserts what was observed rather than what would be preferred.

Concepts

Configured Skip

The gate is disabled unless connection details are supplied through the environment, so the ordinary suite is unaffected and continuous integration minutes are not spent on it. Run it before a release, after upgrading the dependency, and when a write appears to succeed while nothing changes.

Asserting the Observed Behaviour

The assertion records what the system actually does, including where that is defective, together with the instruction to follow when it changes. Asserting the defect turns the gate into a notification: it fails on the day the behaviour is fixed, and the message says what to do about it. Asserting the desired behaviour instead produces a test that fails continuously and is disabled.

Exercising the Real Code Path

The gate calls the project's own client rather than issuing a hand-rolled request. A client reporting 403 Forbidden on deletes that a plain request performs happily is a defect in the client, and a test that issues its own request stays green while the tool tells users records cannot be removed.

Usage Guidelines

Use when:

  • A dependency can fail silently, accepting a write and discarding it
  • A dependency reports an error that is not real
  • An assumption about an external system is load-bearing and unverified

Avoid when:

  • The dependency has a specification and a conformance suite already covering it
  • No instance can be reached from where tests run, and none can be provisioned
  • The behaviour is already covered by an integration test in the ordinary suite

Implementation

from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class ObservedBehaviour:
    """What an external system was measured to do, and what to do when it changes."""

    name: str
    observed: Any
    observed_on: str
    instruction: str

    def message(self, current: Any) -> str:
        """Failure text reporting the change and the action it calls for."""
        return (
            f"{self.name} returned {self.observed!r} on {self.observed_on} and "
            f"now returns {current!r}. {self.instruction}"
        )


class LiveContractGate:
    """Compares recorded external behaviour against a live instance."""

    def __init__(
        self,
        url: str | None,
        token: str | None,
        url_variable: str = "CONTRACT_URL",
        token_variable: str = "CONTRACT_TOKEN",
    ) -> None:
        self._url = url
        self._token = token
        self._url_variable = url_variable
        self._token_variable = token_variable

    @property
    def configured(self) -> bool:
        """Whether an instance is available to run against."""
        return bool(self._url) and bool(self._token)

    def skip_reason(self) -> str:
        """Why the gate is inactive, naming the variables that activate it."""
        return (
            f"set {self._url_variable} and {self._token_variable} to run "
            f"against an instance"
        )

    def check(self, recorded: ObservedBehaviour, current: Any) -> None:
        """Raise AssertionError when the system no longer behaves as recorded."""
        if current != recorded.observed:
            raise AssertionError(recorded.message(current))

Usage

GATE = LiveContractGate(os.environ.get("CONTRACT_URL"), os.environ.get("CONTRACT_TOKEN"))

pytestmark = pytest.mark.skipif(not GATE.configured, reason=GATE.skip_reason())

SAMPLE_TYPES_DISCARDED = ObservedBehaviour(
    name="assay sample types after an API-created assay",
    observed=[],
    observed_on="260806",
    instruction=(
        "The server now returns sample types for an API-created assay. If it "
        "also accepts the relationship, the importer no longer needs sample "
        "types attached by hand; revisit ISAAssayService."
    ),
)


def test_the_server_still_discards_sample_types():
    """The write is accepted, answered 200, and stored nowhere."""
    assay = client.create_assay(sample_types=["dna"])  # our client, not a raw request
    GATE.check(SAMPLE_TYPES_DISCARDED, client.get_assay(assay.id).sample_types)

Trade-offs

Benefits:

  1. Detects silent acceptance and discard, which no mocked test can observe
  2. Notifies when an external defect is fixed, rather than failing until it is
  3. Exercises the project's own client, so client defects surface too

Drawbacks:

  1. Requires a reachable instance and credentials, so it does not run by default
  2. Being skipped by default, it can go unrun for long periods
  3. Writes against a real system need cleanup and can interfere with other users

Real-World Examples

  • Contract tests run against a staging API before a release
  • Provider verification in consumer-driven contract testing
  • Compatibility suites run after upgrading a database or client library

API Reference

design_patterns.gates.live_contract

Live Contract Gate: assert that beliefs about an external system hold.

This is the one gate a mocked suite cannot be. A suite built on doubles verifies the code against its own assumptions, and a wrong assumption is invisible from the inside.

Two things make the pattern work. The gate asserts what was observed rather than what would be preferred, including where the observation is of a defect: that turns the gate into a notification, failing on the day the behaviour is fixed with a message saying what to do about it. And it exercises the project's own client rather than a hand-rolled request, so a defect in the client surfaces too.

LiveContractGate

Compares recorded external behaviour against a live instance.

Example
gate = LiveContractGate(
    os.environ.get("CONTRACT_URL"), os.environ.get("CONTRACT_TOKEN")
)
pytestmark = pytest.mark.skipif(not gate.configured, reason=gate.skip_reason())
Source code in src/design_patterns/gates/live_contract.py
class LiveContractGate:
    """Compares recorded external behaviour against a live instance.

    Example:
        ```python
        gate = LiveContractGate(
            os.environ.get("CONTRACT_URL"), os.environ.get("CONTRACT_TOKEN")
        )
        pytestmark = pytest.mark.skipif(not gate.configured, reason=gate.skip_reason())
        ```
    """

    def __init__(
        self,
        url: str | None,
        token: str | None,
        url_variable: str = "CONTRACT_URL",
        token_variable: str = "CONTRACT_TOKEN",
    ) -> None:
        """Point the gate at an instance, if one was supplied.

        Args:
            url: Base URL of the instance, typically from the environment.
            token: Credential for that instance, typically from the environment.
            url_variable: Name of the environment variable the caller reads
                ``url`` from, quoted in the skip reason.
            token_variable: Name of the environment variable the caller reads
                ``token`` from, quoted in the skip reason.
        """
        self._url = url
        self._token = token
        self._url_variable = url_variable
        self._token_variable = token_variable

    @property
    def configured(self) -> bool:
        """Whether an instance is available to run against.

        An unset environment variable often arrives as an empty string, which
        counts as absent.
        """
        return bool(self._url) and bool(self._token)

    def skip_reason(self) -> str:
        """Explain why the gate is inactive.

        Returns:
            str: Text naming the variables that activate the gate, so a skipped
                run says how to become a real one.
        """
        return (
            f"set {self._url_variable} and {self._token_variable} to run "
            f"against an instance"
        )

    def check(self, recorded: ObservedBehaviour, current: Any) -> None:
        """Assert that the system still behaves as recorded.

        Args:
            recorded: The observation this gate defends.
            current: The value read from the live instance, obtained through the
                project's own client rather than a hand-rolled request.

        Raises:
            AssertionError: If the current value differs from the recorded one.
                The message carries the instruction for what the change means.
        """
        if current != recorded.observed:
            raise AssertionError(recorded.message(current))

configured property

Whether an instance is available to run against.

An unset environment variable often arrives as an empty string, which counts as absent.

__init__(url, token, url_variable='CONTRACT_URL', token_variable='CONTRACT_TOKEN')

Point the gate at an instance, if one was supplied.

Parameters:

Name Type Description Default
url str | None

Base URL of the instance, typically from the environment.

required
token str | None

Credential for that instance, typically from the environment.

required
url_variable str

Name of the environment variable the caller reads url from, quoted in the skip reason.

'CONTRACT_URL'
token_variable str

Name of the environment variable the caller reads token from, quoted in the skip reason.

'CONTRACT_TOKEN'
Source code in src/design_patterns/gates/live_contract.py
def __init__(
    self,
    url: str | None,
    token: str | None,
    url_variable: str = "CONTRACT_URL",
    token_variable: str = "CONTRACT_TOKEN",
) -> None:
    """Point the gate at an instance, if one was supplied.

    Args:
        url: Base URL of the instance, typically from the environment.
        token: Credential for that instance, typically from the environment.
        url_variable: Name of the environment variable the caller reads
            ``url`` from, quoted in the skip reason.
        token_variable: Name of the environment variable the caller reads
            ``token`` from, quoted in the skip reason.
    """
    self._url = url
    self._token = token
    self._url_variable = url_variable
    self._token_variable = token_variable

check(recorded, current)

Assert that the system still behaves as recorded.

Parameters:

Name Type Description Default
recorded ObservedBehaviour

The observation this gate defends.

required
current Any

The value read from the live instance, obtained through the project's own client rather than a hand-rolled request.

required

Raises:

Type Description
AssertionError

If the current value differs from the recorded one. The message carries the instruction for what the change means.

Source code in src/design_patterns/gates/live_contract.py
def check(self, recorded: ObservedBehaviour, current: Any) -> None:
    """Assert that the system still behaves as recorded.

    Args:
        recorded: The observation this gate defends.
        current: The value read from the live instance, obtained through the
            project's own client rather than a hand-rolled request.

    Raises:
        AssertionError: If the current value differs from the recorded one.
            The message carries the instruction for what the change means.
    """
    if current != recorded.observed:
        raise AssertionError(recorded.message(current))

skip_reason()

Explain why the gate is inactive.

Returns:

Name Type Description
str str

Text naming the variables that activate the gate, so a skipped run says how to become a real one.

Source code in src/design_patterns/gates/live_contract.py
def skip_reason(self) -> str:
    """Explain why the gate is inactive.

    Returns:
        str: Text naming the variables that activate the gate, so a skipped
            run says how to become a real one.
    """
    return (
        f"set {self._url_variable} and {self._token_variable} to run "
        f"against an instance"
    )

ObservedBehaviour dataclass

What an external system was measured to do, and what to do when it changes.

Attributes:

Name Type Description
name str

What was measured, phrased so the failure report reads clearly.

observed Any

The value the system actually returned.

observed_on str

Date of the observation in YYMMDD format.

instruction str

The action to take when the behaviour changes.

Example
record = ObservedBehaviour(
    name="assay sample types after an API-created assay",
    observed=[],
    observed_on="260806",
    instruction="The server now returns sample types; revisit the importer.",
)
Source code in src/design_patterns/gates/live_contract.py
@dataclass(frozen=True)
class ObservedBehaviour:
    """What an external system was measured to do, and what to do when it changes.

    Attributes:
        name: What was measured, phrased so the failure report reads clearly.
        observed: The value the system actually returned.
        observed_on: Date of the observation in YYMMDD format.
        instruction: The action to take when the behaviour changes.

    Example:
        ```python
        record = ObservedBehaviour(
            name="assay sample types after an API-created assay",
            observed=[],
            observed_on="260806",
            instruction="The server now returns sample types; revisit the importer.",
        )
        ```
    """

    name: str
    observed: Any
    observed_on: str
    instruction: str

    def message(self, current: Any) -> str:
        """Build the failure text for a change in external behaviour.

        Args:
            current: The value the system returns now.

        Returns:
            str: Text contrasting the recorded value with the current one and
                stating the action the change calls for.
        """
        return (
            f"{self.name} returned {self.observed!r} on {self.observed_on} and "
            f"now returns {current!r}. {self.instruction}"
        )

message(current)

Build the failure text for a change in external behaviour.

Parameters:

Name Type Description Default
current Any

The value the system returns now.

required

Returns:

Name Type Description
str str

Text contrasting the recorded value with the current one and stating the action the change calls for.

Source code in src/design_patterns/gates/live_contract.py
def message(self, current: Any) -> str:
    """Build the failure text for a change in external behaviour.

    Args:
        current: The value the system returns now.

    Returns:
        str: Text contrasting the recorded value with the current one and
            stating the action the change calls for.
    """
    return (
        f"{self.name} returned {self.observed!r} on {self.observed_on} and "
        f"now returns {current!r}. {self.instruction}"
    )