Practice Area
Experimentation and measurement
Douglas Cirillo runs experimentation as a measurement discipline first and a testing program second: tests aimed at a specific outcome, run on data with definitions a whole company can agree on. At Sotheby’s he built the governed measurement foundation the testing ran on. At Lindblad Expeditions, his team worked from a structured testing backlog in which every proposed test carried a goal, a primary metric, and a hypothesis. At A Place for Mom, Optimizely and Amplitude were part of the stack in the platform rebuild he led.
What is experimentation, when it is done right?
Experimentation is how an organization finds out whether a change worked. A test states a hypothesis, names one primary metric before it runs, and reads the result against that metric, which means the tests aim at a specific outcome and the decisioning is built to test against the objective, not the screen.
None of that holds without the layer underneath: events defined the same way by every team, metrics that mean the same thing in every dashboard, and readouts people trust enough to act on. Measurement is the precondition. Shared KPIs, shared language, governance and reporting, agreed before the tooling arguments start.
How I run it
Agree on definitions and objectives first. At Sotheby’s we agreed on what a customer actually was before we argued about tooling. The program defined its core objectives first, then aligned tracking and reporting to them, and opened the resulting dashboards, including the auction reporting, to anyone in the business. Data that only a few technical users can reach cannot settle an argument. The platform was not the strategy. The strategy built the need for the platform.
One schema, governed. Before the Sotheby’s rebuild, tracking lived in multiple analytics platforms and pseudo warehouses running on inconsistent schemas, and teams spent as much time reconciling numbers as reading them. The fix was a single tracking schema, catalogued and governed so unplanned events get flagged before they pollute everything downstream, with the schema aligned to the KPIs the business had defined strategically.
Signals over capture. At Lindblad we did not try to capture everything. We focused on the signals that mattered to bookings, content engagement, lead quality, and journey progression. One useful signal at a time. Phased clarity, not a grand view.
Hypothesis first, one primary metric. Every proposed test in the Lindblad backlog was structured the same way: the goal, the primary metric, the hypothesis, the proposed change, the expected outcome. A test does not run until its primary metric is named. That is what “tests aimed at a specific outcome” means in practice.
Start small, and re-test. Prove value in one place, with one team, before asking the whole organization to believe. Early wins recruit the next team. And when the stack, the model, or the vendor changes, results earned under the old setup do not transfer automatically. So you re-test. That is not a flaw in the approach. That is the approach.
Where I have run it
Sotheby’s. Optimizely ran experimentation on top of the measurement foundation I directed: Twilio Segment as the customer data platform, a single governed schema, and customer interaction data connected to the BigQuery warehouse called Alpine, configured across 23 systems at about 400 million API calls a month. Amplitude came in as the analytics layer, trial first, then enterprise. Tenure outcomes, as that measurement reported them: organic traffic up 167%, search click-through up 48.7%, consignment search placement doubled, consignment ad spend fell 50%, sessions up 90%, unique visitors up 119%. Read the case study.
Lindblad Expeditions. The testing program ran against the booking funnel. Its backlog was an Optimizely working document: every proposed test carried its goal, its primary metric, and its hypothesis, starting with the calls to action on the departure date cards. The funnel itself was instrumented end to end, booking start to completed booking, pre vs post replatform, on the data layer my team built around the core systems migration.
Within about eight months of my start: bookings up 30%, booking funnel completion up 82%, qualified leads up 40%. Read the case study.
A Place for Mom. The stack carried Optimizely for experimentation and Amplitude for analytics, and user testing informed the navigation and content decisions in the rebuild I led. The diagnosis underneath was the settled one: a platform and governance problem, with schema and structure inconsistent from page to page, so quality varied and search engines read the inconsistency. SEO market share rose from 26% to 68%. Read the case study.
A+E Networks. From Gangverk, I directed the History.com replatform with unified event tracking built in, and Amplitude, which A+E already ran, as the measurement layer. A+E adopted the architecture as its migration standard. There was no experimentation program at A+E; the contribution was instrumentation that every following build could share. Read the case study.
The point
Optimizely was in the stack at Sotheby’s, Lindblad, and A Place for Mom. What differed was the problem each program had to solve. What never changed was the order: definitions first, then data, then the test.