Consider the invoice that never arrives: $0.00 from OpenAI, $0.00 from AWS Rekognition, $0.00 from Google Vision. The system those bills would have covered runs a visual search across 30,006 lingerie products, classifies each image across four taxonomies (bra type, panty type, fabric, color), and returns results in under 200 milliseconds. The whole thing runs on a single RTX 3060 Ti.

The deployment described here powers lingeriemuseum.com, a digital lingerie encyclopedia, which needed a feature the large cloud APIs cannot price sensibly at catalog scale: upload a photo, get visually similar products. Run the math on a hosted embedding API and a single re-embedding pass of the catalog costs more than the GPU itself. On-premise inference is the other way. This is the honest version of what the build taught, including what does not work.

Why FashionCLIP, Not Plain CLIP

OpenAI's original CLIP model is general-purpose - trained on 400 million internet image-text pairs, it knows a little about everything. That breadth is the problem when searching a fashion catalog. Ask plain CLIP "is this a bralette or a bustier" and the embedding space puts them close to everything else that looks like an undergarment, because the model was mostly trained on captions like "a person wearing clothing."

FashionCLIP is a CLIP model fine-tuned on 700K fashion image-text pairs from Farfetch. Same 512-dim output, same API, but the embedding space is organized around garment attributes instead of generic object categories. In practice two slip dresses end up nearer each other than either does to a T-shirt dress, which plain CLIP routinely gets wrong.

The model is 600MB on disk, ~1.5GB in VRAM, and loads in about 5 seconds on first request. Lazy-loading it in the FastAPI backend keeps the container starting fast - the model only materializes when someone uses the identify endpoint:

from functools import lru_cache
from transformers import CLIPModel, CLIPProcessor
import torch

@lru_cache(maxsize=1)
def get_fashion_clip():
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model = CLIPModel.from_pretrained("patrickjohncyh/fashion-clip").to(device)
    processor = CLIPProcessor.from_pretrained("patrickjohncyh/fashion-clip")
    model.eval()
    return model, processor, device

def embed_image(pil_image):
    model, processor, device = get_fashion_clip()
    inputs = processor(images=pil_image, return_tensors="pt").to(device)
    with torch.no_grad():
        features = model.get_image_features(**inputs)
        features = features / features.norm(dim=-1, keepdim=True)
    return features.cpu().numpy()[0]  # 512-dim, L2-normalized

That is the entire embedding code. The rest of the system is storage, retrieval, and the classifier heads on top.

The Pose-Aware Trick That Made Search Actually Work

The first version of the search embedded the whole product image. It returned garbage. A photo of a model wearing a bra and panties would match against any photo that had a model in it - because the dominant visual feature was "human body in a studio," not the garment.

The fix was to stop embedding whole images and start embedding body regions. Every incoming photo runs through MediaPipe Pose Landmarker, which detects 33 keypoints on the body and allows the image to be cropped into four regions:

• Top (shoulders to hips) - for bras, bralettes, camisoles
• Bottom (hips to mid-thigh) - for panties, briefs, boyshorts
• Legs (mid-thigh down) - for stockings, hosiery
• Full (full body) - for slip dresses, bodysuits, sets

Each region gets its own FashionCLIP embedding and lives in a separate row in a product_regions table. At query time the uploaded photo goes through the same pose detection, and a bra crop is only compared against top regions, a panty crop only against bottom regions. This one change lifted visual search precision from "looks like it's doing something" to actually usable. The catalog has 30,006 products and 28,006 of them have at least one region embedding - the rest are edge cases where pose detection failed (cropped images, non-human mannequins, studio flat-lay shots).

Storing Vectors in PostgreSQL with pgvector

A lot of teams reach for Pinecone or Weaviate the moment they hear "vector search." That is not necessary here. Postgres with the pgvector extension handles 28K vectors with room to spare, and keeping the vectors in the same database as the product metadata is worth a lot operationally.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE product_regions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    product_id UUID NOT NULL REFERENCES products(id) ON DELETE CASCADE,
    region_type TEXT NOT NULL CHECK (region_type IN ('top','bottom','legs','full')),
    pose_type TEXT,
    crop_path TEXT NOT NULL,
    embedding vector(512) NOT NULL,
    confidence REAL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX product_regions_embedding_idx
    ON product_regions
    USING ivfflat (embedding vector_cosine_ops)
    WITH (lists = 100);

CREATE INDEX product_regions_region_type_idx ON product_regions (region_type);

One detail cost an afternoon: the IVFFlat index has a probes setting that defaults to 1, meaning a query only scans 1% of the index partitions. The results felt random - easy to blame the embeddings for a day before noticing the query planner was literally looking at 1% of the data. Setting probes = 10 at the session level fixed it:

SET ivfflat.probes = 10;

SELECT p.id, p.name, p.brand_id, pr.region_type,
       1 - (pr.embedding <=> $1) AS similarity
FROM product_regions pr
JOIN products p ON p.id = pr.product_id
WHERE pr.region_type = $2
  AND p.is_active = true
ORDER BY pr.embedding <=> $1
LIMIT 18;

The <=> operator is cosine distance. Because embeddings are L2-normalized at write time, cosine distance and inner product give the same ranking, but cosine plays nicer with the index.

Classifier Heads on Top of Frozen Embeddings

Embeddings get you "visually similar." They do not get you "this is a push-up bra with lace trim." For structured classification, small MLP heads sit on top of the frozen FashionCLIP backbone. Three heads so far: bra_type, panty_type, fabric. Each head is a two-layer MLP taking the 512-dim embedding and emitting softmax probabilities over 10-30 classes.

import torch.nn as nn

class TaxonomyHead(nn.Module):
    def __init__(self, embed_dim=512, num_classes=20, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(embed_dim, hidden),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(hidden, num_classes),
        )

    def forward(self, x):
        return self.net(x)

Training data comes from an admin UI where reference images are tagged for each taxonomy term. After 2,100 labeled examples across 89 terms, the accuracies measured on a held-out set:

Classifier Head	Classes	Training examples	Top-1 accuracy
fabric	22	820	82%
bra_type	14	690	73%
panty_type	12	590	57%

The panty classifier is the honest embarrassment in that table. Cheeky underwear sits at 17% accuracy, brief at 36%. The root cause: from most front-facing photos, the structural difference between these categories is genuinely ambiguous - you need the back view, and the catalog rarely has it. No amount of fine-tuning solves a labeling problem that lacks enough signal in the input.

The fix is not more training. It is adding a second image per product (back view) and training the head on the concatenation of front and back embeddings. That is next quarter's work.

What the End-to-End Pipeline Looks Like

The full path from "admin adds a new product" to "product is searchable" runs like this:

1. Admin uploads product image → saved to disk, row inserted in products
2. Every 10 minutes, a cron job on the GPU server runs batch_pose_embed.py --limit 200
3. Script pulls unembedded products, runs MediaPipe Pose, crops into regions
4. Each crop goes through FashionCLIP, producing a 512-dim vector
5. Vectors written to product_regions with region type and confidence
6. Every 20 minutes, batch_gemma4_describe.py generates a structured JSON description via a local 26B Gemma model running in Ollama
7. Classifier heads run lazily on first query, cached thereafter

Query time looks like this:

async def identify(uploaded_image_bytes):
    image = Image.open(io.BytesIO(uploaded_image_bytes))
    regions = crop_to_regions(image)  # MediaPipe pose

    results = []
    for region_type, crop in regions.items():
        embedding = embed_image(crop)
        classifications = classify_with_heads(embedding, region_type)

        similar = await db.fetch_all(
            """
            SELECT p.id, p.name, p.image_url, p.brand_id,
                   1 - (pr.embedding <=> $1::vector) AS similarity
            FROM product_regions pr
            JOIN products p ON p.id = pr.product_id
            WHERE pr.region_type = $2
              AND p.is_active = true
            ORDER BY pr.embedding <=> $1::vector
            LIMIT 18
            """,
            embedding.tolist(), region_type
        )

        results.append({
            "region": region_type,
            "classifications": classifications,
            "matches": similar,
        })

    return results

The final ranking uses a weighted score: 0.70 * visual_similarity + 0.30 * attribute_bonus, where attribute_bonus rewards matches on the taxonomy labels the classifier heads produced. Pure vector similarity alone puts too many visually-similar-but-categorically-wrong items at the top - a black slip dress looks like a black chemise looks like a black nightgown to the embedding space, and the structured labels are needed to separate them.

What This Costs vs the Cloud Alternatives

Here is the comparison run before committing to on-premise. The numbers are measured, not vendor-promised.

Approach	One-time cost	Monthly cost at 30K catalog, 5K queries/day	Latency
OpenAI text-embedding-3-large + GPT-4V for classification	$0	~$380/mo (embeddings + classification calls)	800-1500ms
AWS Rekognition Custom Labels	$0	~$450/mo (training + inference)	400-600ms
FashionCLIP on RTX 3060 Ti (this setup)	~$450 (one-time GPU purchase)	~$15/mo (electricity)	120-200ms

The cloud options pay for themselves about 2.5 months in at this traffic level. At 10x the traffic, the break-even on the GPU is under three weeks. That is before the less tangible wins: no per-query rate limits, no cold-start penalties, no vendor deprecating the model the product was built on, and no customer data leaving the network - which matters for selling into regulated industries.

When On-Premise FashionCLIP Is the Wrong Answer

The honest counter-arguments, because this path should not be followed blindly:

• Your catalog is under 2,000 items. Cloud embedding APIs are trivially cheap at that scale. Use them.
• Your team has no one who can operate a GPU server. The moment NVIDIA drivers break at 2am on a Sunday, someone has to fix them. If that person does not exist, pay the cloud tax.
• Your product categories are outside fashion. FashionCLIP is fine-tuned on clothing. It will be worse than plain CLIP on furniture, food, or electronics. Check for a domain-specific CLIP for your niche first.
• You need multi-modal text+image search from day one. FashionCLIP handles both, but hooking up the text side well takes more care than this covers.
• You need the absolute frontier of accuracy. OpenAI's latest vision models will beat FashionCLIP on edge cases. For 95% of use cases, nobody notices.

What Could Be Done Differently

Three things worth doing from the start:

1. Label more before training, not after. The first panty_type head was trained on 300 examples, followed by three weeks of debugging accuracy. The real fix was labeling another 290. Classifier head accuracy is boringly linear in training data up to about 500 examples per class, and no amount of architectural cleverness fixes a data shortage.

2. Store the raw crop, not just the embedding. Storing only the vector and a crop_box pointer back to the original image meant re-cropping 28K images when a new model variant was added. Storing the actual cropped region on disk costs about 40GB at quality 85 JPEG and saves days when iterating.

3. Measure retrieval quality with a held-out set from day one. Looking at individual results and going "that feels right" wastes time. A tagged set of 200 query-result pairs with human-judged relevance would have caught the probes = 1 bug in an hour instead of a day.

Conclusion

For a catalog of more than a few thousand items, run an actual cost comparison before defaulting to the cloud. The economics of on-premise inference flipped when mid-range consumer GPUs started doing 5-10 ms embeddings on 512-dim models - FashionCLIP is small enough that a 3060 Ti is over-spec'd, leaving headroom for adding more models to the same box. For companies that cannot send customer data to third parties - healthcare, legal, most regulated industries - this is not a cost question, it is the only path. Same code, same architecture, running on a server in your own rack instead of someone else's data center.