The Flight Search Algorithm

# The Flight Search Conundrum: Building Travel Bot with FastAPI, Nuxt, and AI

*An engineering case study on aggregating flight search results, building a flexible web scraping architecture, and using AI to extract structured data from inconsistent web pages.*

## Introduction: The “One Search Is Never Enough” Realization

Finding a flight should be simple. You enter your destination, choose your dates, compare prices, and book your ticket.

In reality, finding the right flight often means jumping between airline websites and travel aggregators, comparing different offers, and trying to make sense of information presented in completely different ways.

I started thinking about how I could simplify this process by building my own flight search application.

That idea became **Travel Bot**, a full-stack project that brings together asynchronous Python, modern Vue development, web scraping, and AI-assisted data extraction.

My goal wasn't simply to build another search interface. I wanted to explore how to coordinate multiple data sources, normalize inconsistent information, and deliver results through an architecture that could evolve as I added new providers.

The Architecture:

I designed Travel Bot around two main services: a backend responsible for search orchestration and data extraction, and a frontend responsible for the user experience.

The backend uses FastAPI and Python 3.13, while the frontend uses Nuxt 4, Vue 3, and TypeScript. Docker Compose brings both services together in a local development environment.

The main architectural components are:

- **Search orchestration:** Coordinates searches across supported sources.
- **Scraping providers:** Retrieve web content through Spider Cloud or Playwright.
- **Parsing service:** Extracts flight information and maps it into structured results.
- **AI parser:** Provides an optional fallback when rule-based extraction cannot identify fares.
- **Frontend application:** Provides airport autocomplete, search functionality, and the interface for presenting results.

I separated the scraping providers from the search orchestration logic so that the application would not depend on a single retrieval implementation.

This abstraction gives me flexibility to experiment with different providers without having to redesign the entire search workflow every time I change how a page is retrieved.

It also creates a clearer separation of responsibilities: retrieving content is one problem; interpreting that content is another.

The Scraping Conundrum: Spider vs. Playwright

One of the first challenges was deciding how to retrieve content from different flight search sources.

Not every website behaves the same way. Some pages expose useful information in their initial HTML, while others rely heavily on JavaScript to render their content.

I wanted Travel Bot to support more than one approach.

### Spider Cloud

Spider provides a cloud-based scraping service with JavaScript rendering capabilities.

It offers a relatively straightforward way to retrieve page content without managing the browser execution environment locally. The trade-off is that the application depends on an external service, API configuration, and the provider's usage limits and pricing.

### Playwright

Playwright provides browser automation and more direct control over browser behavior.

Because it can run locally, it offers an alternative to cloud-based scraping without requiring a Spider API key. However, it introduces additional setup and browser-management considerations.

### Why support both?

Rather than choosing one implementation and tightly coupling it to the application, I built the scraping layer around a provider-agnostic abstraction.

This allows me to configure the provider through an environment variable:

SCRAPING_PROVIDER=spider

Or switch to:

SCRAPING_PROVIDER=playwright

The objective is to keep provider-specific concerns isolated while maintaining a consistent search workflow.

This design also makes future experimentation easier: adding a provider should primarily involve implementing its integration rather than rewriting the entire search service.

Extracting Structured Flight Data

Retrieving a webpage is only half the problem.

The next challenge is turning inconsistent page content into structured flight options that the frontend can consume.

Different websites represent prices, airlines, and offers in different ways. A parser that works for one page structure may fail completely on another.

I addressed this by organizing the parsing process into two stages.

### Stage 1: Rule-based parsing

The `ScrapingService` attempts to identify flight fares using regular expressions and known content patterns.

The current parsing logic includes patterns for:

- Playwright row blocks.
- Google Flights price cards.
- Aggregator offer snippets.
- Known airline matching through a carrier allowlist.

The allowlist is maintained in `carriers.py`, while the primary parsing and search orchestration logic lives in `services/scraping/service.py`.

I wanted deterministic parsing to be the first approach because known patterns can be handled without introducing an AI dependency into every search.

However, rule-based parsing has an obvious limitation: it depends on the structures and formats the application knows how to recognize.

When websites change their markup or introduce unfamiliar representations, those assumptions may no longer hold.

That led to the next part of the architecture.

Adding AI as a Fallback

Instead of relying exclusively on regular expressions, I integrated Google Gemini as an optional fallback parser.

The idea is straightforward: if the rule-based parser cannot identify any fares, but the retrieved content contains signals that prices may be present, the application can attempt AI-assisted extraction.

The `AIParser`, implemented in `services/scraping/ai_parser.py`, sends a trimmed excerpt of the scraped content to Gemini and maps the returned JSON into structured `TravelOption` records.

The configuration uses `google-genai` and supports an optional model setting:

```bash
GOOGLE_API_KEY=your_google_api_key_here
GEMINI_MODEL=gemini-3.1-flash-lite
```

If the API key is not configured, the application can still use its rule-based parsing path.

I like this approach because it gives each technique a defined responsibility:

- Regular expressions handle known patterns.
- AI assists with content that the existing rules cannot parse.
- Structured application models provide a consistent representation for downstream processing.

AI is not a replacement for validation, though. A model can misinterpret a price, confuse unrelated numbers, or return incomplete information. Extracted results still need appropriate validation, and a successful parse should not be confused with independently verified fare availability.

This is an area where I want to continue improving reliability as the project evolves.

Searching Multiple Sources

Searching multiple providers introduces another challenge: each source can take a different amount of time to respond.

If one provider is slow, waiting for every other provider to finish before showing any results can make the entire experience feel slower.

Travel Bot uses asynchronous Python and AnyIO to coordinate its search operations. The project also includes a streaming endpoint built around Server-Sent Events (SSE).

The backend exposes two search endpoints:

```text
POST /api/search_travel
POST /api/search_travel/stream
```

The first returns search results through a standard API request. The second streams results as individual sources finish.

This creates the foundation for progressive result delivery: instead of treating the entire search as one indivisible operation, the application can communicate results as they become available.

AnyIO also provides tools for structured concurrency and timeouts, which are important when coordinating independent operations.

There are still trade-offs to consider. A provider may fail, return incomplete content, or take longer than expected. A robust search workflow needs to handle those cases without allowing one unreliable source to compromise the entire experience.

For me, this is one of the most interesting aspects of the project because it connects backend execution strategy directly to frontend responsiveness.

## What This Project Has Taught Me

Building Travel Bot has given me an opportunity to work through several engineering problems that are easy to underestimate when looking at a feature list.

**Abstraction is useful when it protects against real changes.** Supporting Spider and Playwright behind a provider-agnostic interface gives the application flexibility without forcing the search orchestration to know every implementation detail.

**AI works best when it has a defined role.** Using Gemini as a fallback rather than the default parser gives the system a clear strategy for combining deterministic rules with AI-assisted extraction.

**Concurrency is a user experience decision.** Coordinating independent search sources and streaming their results can make the application more responsive, provided that failures and timeouts are handled appropriately.

**Deployment configuration is part of application architecture.** The distinction between browser-side and server-side API requests demonstrates how networking and runtime environments influence full-stack application behavior.

**Structured data matters as much as data retrieval.** Fetching a page does not automatically produce a reliable flight offer. Extraction, validation, normalization, and error handling all matter.

These are the kinds of problems I enjoy exploring because they require thinking across the frontend, backend, infrastructure, and integration boundaries.

## What's Next? The Reliability Experiment

Travel Bot is an ongoing project, and there are several areas I want to improve as development continues.

My next priorities include:

- **Parser reliability:** Expand test coverage for known page patterns, malformed content, and unexpected price formats.
- **Provider resilience:** Improve timeout handling, partial failure behavior, and error reporting.
- **Data validation:** Strengthen validation of extracted flight information before presenting it as a usable option.
- **Search performance:** Measure response times across providers and identify bottlenecks.
- **Streaming experience:** Refine how the interface displays partial results and communicates unsuccessful searches.
- **Observability:** Track provider failures, extraction outcomes, and search latency to make debugging easier.

I also want to evaluate extraction accuracy against known examples instead of assuming that successfully parsing a response means the extracted information is correct.

Those measurements will help guide future architectural decisions and show where additional complexity is actually worthwhile.

## Final Thoughts: The Flight Search Conundrum Continues

Travel Bot started with a straightforward idea: make it easier to discover flight options without manually searching multiple websites.

Implementing that idea led me into a much broader set of engineering challenges involving asynchronous execution, provider abstractions, web scraping, AI-assisted parsing, container networking, and CI/CD.

The project brings together technologies I enjoy working with—Python, FastAPI, Vue, Nuxt, and TypeScript—while giving me a practical way to explore how they interact in a real application.

I am particularly interested in the boundary between structured software logic and unstructured web content: knowing when traditional parsing is sufficient, when AI can help, and how to validate the result.

There is still work to do, especially around reliability and automated testing, but that is part of what makes the project worthwhile.

After all, the interesting part of building software is rarely just getting the happy path to work. It's figuring out what happens when the real world refuses to follow your assumptions.

**Want to explore the implementation?**

Check out the [Travel Bot repository on GitHub](https://github.com/charlieweb/travel-bot-flights) to explore the code, follow the development, and see how the architecture evolves.

*Built with Python, FastAPI, AnyIO, Nuxt 4, Vue 3, TypeScript, Pinia, Tailwind CSS, Playwright, Spider, Google Gemini, AirLabs, Docker, GitHub Actions, and Railway.*