Memegician: Agentic AI Pipeline for Prompt‑to‑Meme Media Generation

Challenge

General-purpose image models are great at pixels but terrible at memes. They mangle text, ignore the cultural context of templates, and can't reliably place captions inside the panels that fans actually expect. We needed a system that thinks the way a person making a meme thinks: pick the right template for the joke, write captions that land, and place them exactly where they belong — fast enough to feel like a chat reply.

That's the product we built and launched as Memegician: an enterprise-ready SaaS that translates a single natural-language prompt into perfectly contextualized memes and captioned reaction GIFs.

Solution

Memegician is not a single model call — it's a multi-step agentic workflow where each stage has a narrow job and a strict contract with the next one.

• Semantic expansion & vector search over templates. A planner LLM rewrites the raw user prompt into a structured emotional intent (situation, target, tone, format) and a small set of embedding queries. Those queries hit a pgvector index of template embeddings built from each template's canonical caption, tags, and historical usage. We fuse cosine similarity with format constraints (still vs. GIF, panel count, aspect ratio) to shortlist candidates.
• Structured LLM orchestration for captions and coordinates. The shortlist plus the template's panel schema is handed to a high-performance model (GPT-4o for text-dominant memes, Gemini 1.5 for multi-panel reasoning). The model is forced through a JSON Schema that demands per-panel text, font weight, alignment, and pixel-space coordinates that fit inside each panel's bounding box. A repair step catches over-long captions and re-prompts before anything is rendered.
• Dynamic canvas-based media rendering. Validated output flows into a high-speed canvas engine that composites transparent template assets, draws captions with stroke and shadow for legibility, and exports either a static PNG/JPG or an animated GIF (per-frame text drawing with an in-house encoder). Brand kits — fonts, palette, watermark — are applied at this stage so the same pipeline serves consumer and enterprise tenants from one path.
• Evaluation & guardrails. A golden set of prompt → expected-template pairs runs in CI on every prompt or schema change. Caption fit is measured by re-projecting rendered text back through OCR and comparing it to the structured output, which catches truncation and overflow before users do. Safety classifiers gate prompts for hate, harassment, and trademark issues.
• Observability. Every generation captures the planner trace, retrieved candidates, structured output, render parameters, and final asset URL so any output can be replayed and diffed end-to-end.

Results

• A single natural-language prompt produces a brandable meme or animated GIF in under 5 seconds.
• 2,000+ templates indexed and selectable through semantic search, with new templates onboarded in minutes.
• 250k+ generations served since launch with 96% caption-fit accuracy on the held-out eval set.
• One pipeline serves the public SaaS at memegician.ai and white-label enterprise tenants with their own brand kits.

Technical Highlights

• Planner → retriever → generator → renderer agentic graph with strict JSON-Schema contracts between every stage.
• pgvector template index with hybrid scoring across semantic intent and format constraints.
• Structured Outputs on GPT-4o and Gemini 1.5 producing per-panel text and pixel-space coordinates in one call.
• Canvas-based static and animated GIF rendering with brand-kit theming and watermarking.
• OCR-based caption-fit evaluation and safety classifiers wired into CI and into the live request path.

See it live at memegician.ai.