Extract a URL to Clean Markdown for RAG in Pipedream
Point a Pipedream workflow at a URL and let Rendex return just the article — clean, reader-mode Markdown with the nav, ads, cookie banners, and boilerplate stripped out. One code step turns a messy page into ingestion-ready text for your embeddings or LLM step.
Last updated 2026-07-15
The Problem
RAG pipelines live or die on clean input, but the open web is hostile to it: raw HTML is buried under navigation, cookie banners, ad slots, comment threads, and social widgets. Feeding that to an embedding model wastes tokens and pollutes retrieval with junk. The usual fixes — a Readability library you host and babysit, or a headless browser to render JS-heavy pages first — are exactly the infrastructure a no-code Pipedream workflow is meant to avoid, and they still leave you writing the extraction logic.
The Solution
Add one Rendex code step. POST the URL to /v1/extract and Rendex renders the page with a real browser, runs reader-mode extraction, and returns clean Markdown plus the title, byline, and excerpt — the JS-heavy pages a plain fetch can't read included. Hand result.data.content to your OpenAI embedding step and upsert into Pinecone, pgvector, or Weaviate. No HTML parsing, no boilerplate rules to maintain, no browser to host — and extraction only costs a credit when it actually returns content (failures are refunded).
How the Workflow Runs
New URL
An RSS item, a webhook payload, or a scheduled list surfaces a page to ingest.
Rendex extract
A code step POSTs the URL to /v1/extract and gets back clean reader-mode Markdown.
Embed / store
Pass the Markdown to an OpenAI embedding step and upsert into Pinecone, pgvector, or your store.
Input → Rendered Output
Rendered by Rendex
What You Need
- A Rendex API key — the free tier includes 100 extractions/month with no card required.
- A Pipedream account and a workflow with a trigger (RSS, HTTP/webhook, or Schedule).
- A downstream step: an OpenAI (or other) embedding action and a vector store (Pinecone, pgvector, Weaviate) — or just an LLM summarize step.
- The URL to ingest available on the trigger step (steps.trigger.event.url or similar).
What This Recipe Uses
Clean Markdown, not raw HTML
Rendex renders the page and runs reader-mode extraction, returning just the article as Markdown — nav, ads, cookie banners, and comment threads stripped out.
Ingestion-ready for LLMs
Markdown is the format models read best, and you get title, byline, and excerpt alongside it — ready-made metadata for your chunks and citations.
Reads JS-heavy pages
A plain fetch returns empty on SPA and JS-rendered pages. Rendex renders with a real browser first, so the extracted text is the content a reader actually sees.
Failures are refunded
Extraction costs one credit only when it returns content — a dead link, timeout, or unreadable page refunds the credit, so a bad URL in the feed never bills you.
Build It
// Pipedream Node code step. Reads the URL from the trigger, returns Markdown
// to the next step (an OpenAI embedding action, then a vector-store upsert).
export default defineComponent({
props: {
rendexApiKey: { type: "string", secret: true },
},
async run({ steps }) {
const url = steps.trigger.event.url; // e.g. a new RSS item's link
const res = await fetch("https://api.rendex.dev/v1/extract", {
method: "POST",
headers: {
Authorization: `Bearer ${this.rendexApiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ url, extractFormat: "markdown" }),
});
if (!res.ok) throw new Error(`Rendex extract failed: ${res.status}`);
const { data } = await res.json();
// -> hand data.content to your embedding step; keep title/byline as metadata.
return { markdown: data.content, title: data.title, byline: data.byline };
},
});curl -X POST https://api.rendex.dev/v1/extract \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "url": "https://example.com/blog/post", "extractFormat": "markdown" }'
# -> { "data": { "content": "# Title\n\nClean article text...",
# "title": "...", "byline": "...", "excerpt": "...", "length": 4213 } }100 free API calls/month — no credit card required. Get your API key and start building.
Reach for this whenever a workflow needs the *meaning* of a page, not a picture of it — building a knowledge base from a documentation site, summarizing every new article in an RSS feed, enriching a lead from its company blog, or keeping a vector store fresh as pages change. The hard part of RAG ingestion isn't the embedding call, it's getting clean text in front of it, and the open web makes that miserable: nav, ads, cookie walls, and JS-rendered bodies that a plain HTTP fetch returns empty. Rendex's extract endpoint solves it in one step — a real browser renders the page, reader-mode pulls out the article, and you get Markdown a model can actually use, with the title and byline as metadata for your chunks. In Pipedream it's a short code step between the trigger and your embedding/store step, so the whole ingestion path stays on the canvas with nothing to deploy. It runs on the free tier, and because a failed extraction refunds its credit, a dead link never costs you.