2026 - Present
Private client workAI Property Pipeline
Replacing manual real-estate data entry with a Claude-powered pipeline that runs while the team sleeps, scrape, extract, validate, deliver.
Senior Full-Stack DevOps Engineer · AI systems
- property data entry replaced by AI ingestion
- Manual → auto
- scrape → extract → validate → upload, per listing
- Hands-free
- every AI output validated before it touches the DB
- Schema-valid
- human review path in the admin console
- Audit-logged
The Problem
A real-estate marketplace lives and dies by listing data quality, and that data arrives as unstructured chaos: agent emails, scraped listings, 図面 floor-plan files, inconsistent room layouts, Japanese addresses with multiple valid formats, prices quoted five different ways.
Staff were re-typing this into the platform by hand (slow, error-prone, and expensive) and every new listing repeated the same manual workflow: find the source, download the 図面, read it, type it in, upload it.
The interesting engineering problem wasn't 'call an LLM.' It was: how do you put a probabilistic model inside a data pipeline that a business depends on, without letting hallucinated square footage into a production database?
The Constraints
Wrong data is worse than no data
A hallucinated rent price on a live listing is a business incident, not a bug. The pipeline needed hard validation boundaries: AI extracts, code verifies, humans arbitrate the ambiguous cases.
Japanese real-estate formats
Layouts like 2LDK, area in ㎡ and tsubo, addresses in kanji with ward/chōme structure, extraction had to normalize domain-specific Japanese conventions, not just 'read text.'
Token costs at pipeline scale
Running frontier models over every document, every day, adds up. Prompt design, batching, and choosing when not to call the model were cost-engineering decisions, made with the same bill-as-telemetry discipline that had already caught a leaked API key on this platform costing ¥100,000/month.
Small team, no ML infrastructure
No data engineers, no ML platform. Whatever I built had to run on the existing stack (Python, PostgreSQL, Render) and be operable by a generalist team.
The Architecture
A validation-first pipeline: Claude does the messy cognitive work of extraction, but nothing it says is trusted until it passes schema validation, domain rules, and confidence gating. Low-confidence extractions route to humans in the audit-logged admin console rather than silently guessing.
- The full chain is hands-free: scrape the source site, download the 図面 floor-plan file, extract and validate the data, post the result to Slack for the team, and upload it to the management site, work that used to be an entire manual workflow per listing.
- Extraction prompts demand structured output against a fixed schema; responses that fail validation are retried with the error context, then routed to review, the model is never allowed to 'probably' write to the database.
- Token spend is engineered, not accepted: prompts are scoped per document type, calls are batched, and the model is skipped when deterministic parsing suffices, the same bill-as-telemetry discipline that caught the leaked API key.
- Every automated write is attributable and reversible via the audit log, the same admin console humans use, so AI and human edits share one review model.
The Trade-offs
Every architecture is a set of bets. These are the ones I made, and what I gave up to make them.
Extraction approach
Regex parsers shatter on the long tail of agent formatting. Claude handles the variability; the schema layer restores determinism. The combination is more robust than either alone, and new source formats cost a prompt tweak, not a new parser.
Failure handling
100% automation was possible and wrong. Routing the uncertain 10% to humans keeps the error rate near zero while still eliminating ~90% of the manual work, and the review queue doubles as a labeled dataset for improving prompts.
Team visibility
The team lives in Slack. Posting each processed listing where the team already works builds trust in the automation and surfaces failures in minutes, a silent job would have been 'working' right up until someone noticed a month of bad data.
Runtime
The pipeline is I/O-bound API orchestration, not model training. Running it as plain scheduled jobs on infrastructure the team already operates kept the system understandable by everyone, not just me.
The Outcome
The pipeline runs in production and changed how the business operates day to day.
- Property data ingestion that previously consumed staff hours per listing now runs automatically, with humans handling only the flagged edge cases.
- New listing sources are onboarded with a prompt and configuration change rather than a hand-written parser, so coverage keeps growing without rewrites.
- The validation-first pattern became the template for AI features across the company, including the AI-assisted code review I introduced into our pull-request workflow.
- Built in collaboration with the CEO, agents, photographers, and SEO specialists, the pipeline serves real workflows, not a tech demo.
Lessons Learned
Treat the model as an untrusted client
The architecture that works is the one where the LLM has no write access to anything. It proposes; typed validation disposes. Every production AI system I build now starts from that trust boundary.
The last 10% shouldn't be automated
Chasing full automation would have traded a near-zero error rate for marginal savings. Knowing where to stop automating is as much an engineering decision as knowing how to automate.
AI features are pipelines, not prompts
The prompt was maybe 5% of the work. Normalization, validation, retries, cost control, review queues, and observability were the other 95%, classic data engineering, which is exactly why experienced engineers ship better AI systems.