A while back I tried building a small news dashboard for myself. The plan was simple: pull stories from about a dozen sites, show them on one page, done. I started with RSS feeds and a bit of scraping, and it worked fine for about two weeks.
Then things started breaking. One site changed its HTML and my parser returned empty titles. Another dropped its RSS feed completely. A third started blocking my requests. I spent more time fixing the scraper than reading the news.
That is the real problem with DIY aggregation. It works at small scale, but every new source adds another thing that can break. A news aggregation API takes that whole layer off your plate. You send one request, you get clean JSON back, and the provider deals with the thousands of sources behind it.
In this tutorial we will build a working news aggregator in Python. It will fetch headlines, run keyword searches, remove duplicates, store everything in SQLite, and show the results in a simple Flask page. By the end you will have something you can actually run and extend.
What a News Aggregator Actually Needs
Before writing code, it helps to know what makes an aggregator useful instead of just noisy.
Sources, deduplication, categories and freshness
Sources. More sources means better coverage, but only if they are reliable. You want major publishers and local outlets, not just the same five big sites.
Deduplication. The same story often appears on many sites, sometimes word for word through wire services. Without deduplication, your feed turns into the same headline repeated ten times.
Categories. Readers want to jump straight to tech, business or sports. Your data needs category labels you can filter on.
Freshness. A news feed that is six hours behind is not really a news feed. New articles should show up quickly after they are published.
If you build all of this yourself, you are maintaining crawlers, parsers and a classification system. With an API, most of it comes built in.
Choosing a News API for Aggregation
For this project I am using AllNewsAPI, a news API built for developers. A few things made it a good fit for an aggregator.
Coverage
It pulls from over 250,000 publishers across 196 countries, in 22 languages, sorted into 29 categories. That is enough range to build a global feed or a very local one, depending on your filters.
Search filters
You can search by keyword or exact phrase, use Boolean operators like AND, OR and NOT, filter by publisher, country, language, or category, and limit results to a date range. For an aggregator this matters a lot, because good filtering is what separates a useful feed from a messy one.
There is also a free plan with 50 requests per day, which is plenty to build and test everything in this guide. Note that the free plan is meant for personal projects and testing, so you would move to a paid plan before launching anything commercial.
Step 1: Get an API Key and Make Your First Request
Create a free account on AllNewsAPI and copy your API key from the dashboard. No credit card is needed.
Store the key as an environment variable rather than pasting it into your code:
export ALLNEWSAPI_KEY=”your_key_here”
pip install requests flask
Now make a first request to the search endpoint:
import os
import requests
API_KEY = os.getenv(“ALLNEWSAPI_KEY”)
BASE_URL = “https://api.allnewsapi.com”
response = requests. get(
f”{BASE_URL}/search”,
params={“apikey”: API_KEY, “q”: “electric vehicles”, “lang”: “en”, “max”: 5},
timeout=10,
)
response.raise_for_status()
data = response.json()
print(“Total matches:”, data[“totalArticles”])
for article in data[“articles”]:
print(article[“title”], “|”, article[“source”] [“name”])
If you see a list of titles with publisher names, everything is connected. Each article comes back with fields like title, description, URL, image, publishedAt, category, and source, which is everything an aggregator card needs.
Step 2: Fetch Top Headlines by Category and Country
Search is great for specific topics, but the front page of an aggregator usually shows top stories. For that, use the headlines endpoint.
def get_headlines(category, country=”us”, max_results=5):
response = requests. get(
f”{BASE_URL}/headlines”,
params={
“apikey”: API_KEY,
“category”: category,
“country”: country,
“lang”: “en”,
“max”: max_results,
},
timeout=10,
)
response.raise_for_status()
return response.json().get(“articles”, [])
tech_news = get_headlines(“technology”)
business_news = get_headlines(“business”, country=”gb”)
You can pass several countries at once, like country=”us,gb,ca,” which is handy if you want an English language feed from multiple markets in a single call.
Step 3: Add Keyword Search with Boolean Operators
This is where an aggregator starts to feel personal. Say you follow AI funding news but you are tired of hearing about one particular company. You can shape the query exactly:
def search_news(query, max_results=5, sort_by=”publishedAt”):
response = requests. get(
f”{BASE_URL}/search”,
params={
“apikey”: API_KEY,
“q”: query,
“lang”: “en”,
“max”: max_results,
“sortby”: sort_by,
},
timeout=10,
)
response.raise_for_status()
return response.json().get(“articles”, [])
ai_funding = search_news(‘(startup OR startups) AND “series a” AND AI’)
crypto_no_btc = search_news(“cryptocurrency NOT bitcoin”)
Setting sortby to relevance instead of publishedAt is useful for topic pages, where the best match matters more than the newest one.
Step 4: Remove Duplicates and Store Results (SQLite)
Now we need somewhere to keep the articles, and a way to stop the same story from showing up twice. SQLite is perfect for this because it needs zero setup.
I use two checks. The URL is the primary key, so the exact same link can never be saved twice. I also store a simple fingerprint of the title, which catches the same wire story published on different sites.
import sqlite3
import hashlib
import re
conn = sqlite3.connect(“news.db”)
conn.execute(“””
CREATE TABLE IF NOT EXISTS articles (
url TEXT PRIMARY KEY,
title TEXT,
title_hash TEXT UNIQUE,
description TEXT,
source TEXT,
category TEXT,
image TEXT,
published_at TEXT
)
“””)
def title_fingerprint(title):
cleaned = re.sub (r”[^a-z0-9 ]”, “”, title.lower())
return hashlib.md5(cleaned.strip().encode()).hexdigest()
def save_articles(articles):
for an article:
conn.execute(
“INSERT OR IGNORE INTO articles VALUES (?, ?, ?, ?, ?, ?, ?, ?)”,
(
a[“url”],
a[“title”],
title_fingerprint(a[“title”]),
a.get(“description”),
a[“source”][“name”],
a.get(“category”),
a.get(“image”),
a[“publishedAt”],
),
)
conn.commit()
INSERT OR IGNORE quietly skips anything that clashes with an existing URL or title fingerprint. It is not perfect, since two outlets can rewrite the same story with different headlines, but it removes most of the obvious repeats.
Step 5: Build a Simple Flask Front End
Last piece: a page to actually read the news. Flask keeps this very short.
from flask import Flask, render_template_string
app = Flask(__name__)
PAGE = “” “
<h1>My News Feed</h1>
{% for a in articles %}
<div style=”margin-bottom:20px”>
<a href=”{{ a[0] }}” target=”_blank”><strong>{{ a[1] }}</strong></a>
<p>{{ a[2] or “” }} </p>
<small>{{ a[3] }} | {{ a[4] }} </small>
</div>
{% endfor %}
“””
@app.route(“/”)
def home():
db = sqlite3.connect(“news.db”)
rows = db.execute(
“SELECT url, title, description, source, published_at “
“FROM articles ORDER BY published_at DESC LIMIT 50”
).fetchall()
return render_template_string(PAGE, articles=rows)
if __name__ == “__main__”:
app.run(debug=True)
Put the fetching code in a separate script, something like refresh.py, and run it on a schedule with cron. Then start the Flask app and open localhost:5000. You now have a working aggregator.
Scaling Tips: Rate Limits, Caching and Pagination
A few lessons that will save you headaches once the project grows.
Plan your request budget. On the free plan you get 50 requests per day. If you pull four categories every two hours, that is 48 requests, which fits nicely. Pulling every 15 minutes will not. Counters reset at 00:00 UTC, and once you hit the limit, the API returns a 403 error, so handle that in your code instead of letting the script crash.
Cache on your side. Your users should read from SQLite, never directly from the API. The database is your cache. That way a traffic spike on your site does not burn through your request quota.
Use pagination for backfills. Each response includes currentPage and nextPage. Loop through pages with the page parameter until nextPage comes back as null. Paid plans also let you raise max so each request returns more articles.
Search the right fields. The attributes parameter controls whether your keywords are matched against the title, the description or the full content. Matching on the title only gives tighter, more precise results.
If you want the full list of parameters, read the documentation. It has examples in Python, JavaScript, Go and several other languages.
Bonus: Let an AI Agent Query Your Aggregator via MCP
Once your feed is running, there is a fun extension. AllNewsAPI also offers an MCP server, which lets AI assistants like Claude, Cursor or Copilot search news directly. The MCP server is powered by the same news API you just used, so it runs on your existing API key.
Adding it to an MCP client takes one small config block:
{
“mcpServers”: {
“allnewsapi”: {
“type”: “remote”,
“url”: “https://mcp.allnewsapi.com/mcp”,
“headers”: { “X-API-Key”: “YOUR_API_KEY” }
}
}
}
After that, you can ask your assistant things like “what are the top tech headlines in Germany today” and it will call the news API behind the scenes. It is a nice way to test queries in plain English before adding them to your aggregator.
FAQs
What is a news aggregation API?
It is a service that collects articles from thousands of publishers and gives you access to all of them through one API. Instead of scraping sites one by one, you send a single request and get structured data back.
Can I build a news aggregator for free?
Yes, for learning and personal use. The free plan covers 50 requests per day, which is enough to build and test the project in this guide. For a commercial product you will need a paid plan.
Is it legal to display news from an API on my site?
Showing headlines, short descriptions and links back to the original article is the standard approach. Republishing full articles as your own can cause copyright problems, so always credit the publisher and link to the source.
How often does news API data update?
AllNewsAPI adds new articles to its database continuously as publishers release them, and headlines are updated in real time. How fresh your own feed feels depends on how often your refresh script runs.
Wrapping Up
With a few Python files, you now have an aggregator that fetches headlines, searches by topic, filters out duplicates and serves everything from a local database. No crawlers to babysit, no broken parsers on Monday morning.
From here you could add user accounts, email digests, or topic pages for different interests. If you want to try it yourself, grab a free key from AllNewsAPI and have your first feed running in an afternoon.
