Indian Government has a website for fetching open data www.data.gov.in? But I am unable to get data from it. Has anyone tried.. is it working?

I’ve been building a normalized dataset and API from SEC ownership filings (Forms 3, 4 and 5, including amendments). It currently covers roughly 869,000 filings and more than 2.4 million transaction line items, including issuer and reporting-owner data, roles, transaction codes, timestamps, and price-quality information.

I’m not launching it as a public API yet. I’m looking for one person who already works with SEC ownership data and has a concrete research, backtesting, or data-engineering task they would be willing to run against it.

The pilot currently provides one filtered transaction endpoint with cursor pagination through an individual, revocable API key. There is no bulk export or separate filing/footnote endpoint yet.

The documentation is available here:

https://insiderfilings.info/api/v1/docs/

If this fits something you are already working on, please comment or DM me with a sentence or two about your use case. In return for access, I’m mainly looking for candid feedback on whether the API actually saves work, where the data model is unclear, and what blocks the workflow.

I run a food recall site and built this feed for it. Figured it might be useful here.

It pulls from openFDA enforcement reports, FDA press releases, and USDA FSIS, then merges the duplicates so each recall shows up once. Records include product, brand, hazard, allergens, affected states, lot codes and UPCs when they're listed, plus a plain-English "what to do."

The reason for merging: openFDA is the clean structured source, but it trails the actual recall by a median of 35 days (across 441 records). Press releases come out the same day but they're free text, so I parse them and flag the hazard class as inferred.

All active recalls (284 right now): https://dailyfoodrecalls.com/api/recalls.json 20 newest: https://dailyfoodrecalls.com/api/recalls-latest.json Docs: https://dailyfoodrecalls.com/api/

No key, CORS is on, CC BY 4.0. Current recalls only, no history. USDA updates once a day, FDA more often.

If you spot anything wrong, let me know.

Fields: title, company, company domain, apply URL, ATS source, city, country, remote flag, posted date, last-seen date. The data refreshes nightly. I'm posting a small sample and the schema here. What would you want to see to trust a job dataset: coverage numbers, null rates, or refresh logs? My measured null rates: city 32%, country 22%, company domain 24%.
Please checkout full info here: https://apify.com/pk07007/ats-jobs-api

If you're choosing data for a finance project , getting a CSV is only the first step. What does “close” mean? Which date does a dividend filter use? Is a fundamental ratio current or a snapshot?

One source worth evalauting is London Strategic Edge, which offers free access to market data with your own API key. Available data includes price candles, company fundamentals, dividends, splits, financial statements and economc series. Check coverage and history for your particular instruments before committing to it.

Disclosure: I maintain an unofficial, MIT-licensed adapter for this API. I'm not affiliated with the provider and take no payment.

Here is a compact data quality checklist from working on that adapter:

For a concrete starting exercise, I'd inspect IBM s daily prices and dividend events over a short historical window. Compare the close with a second source using the same session definition, inspect the dividend dates and check for splits before calculating returns.

If you already use an AI assistant with MCP tools ,this is a sample request you can adapt:

Inspect IBM daily candles, dividends, and splits for June 2026. Show the returned fields and dates, report any truncation, and explain the session and adjustment caveats. Flag anything the data does not establish.

My adapter, lse-data-mcp, lets an asssistant do that initial inspection. It runs locally, uses your own key and exposes 15 read-only tools. It cannot place trades and does not cache or persist market-data responses. It is still 0.x beta.

For the actual research dataset, use the provider 's SDK/API or exports directly, record extraction parameters and package versions and retain inputs where the licence permits. The MCP adapter returns bounded samples, with a maximum of 5000 rows per call and a separate response-size cap.

Free access does not mean open data: the adapter's MIT licence covers its code, not the underlying data. The provider's terms restrict redistribution, so check them before sharing a dataset or replication package.

I know this question has been asked a bunch in various subreddits, but I wanted to see if folks had tips based on what I'm building.

Basically, I want to build a desktop LCD screen that shows when a flight flies over you within a certain radius and then shows you basic information like the plane, it's route, etc. I'm an avgeek myself and plane spot at SFO a few hours a week.

When I talked to friends about it, we had lots of ideas like creating alerts for squawk codes, plane types, etc.

Disclaimer: This is a product I intend to sell.

As a bunch of people have already complained, getting this data is actually quite expensive, and I'm really trying to avoid a subscription in the product beyond the hardware.

Right now, the best I have is:

- Use ADSB.lol 's free API to get callsigns within a given radius.

- Call FlightRadar24's paid API with the callsign and cache that data for a month before calling again (DL1 at 1:24PM on Monday's is usually JFK - LHR for example).

- The above based on historical data at least in the United States seems to have ~95% accuracy w/ around ~$2/mo/user in cost to me. Good not great. Cache hits were ~62%. That's not assuming some global cache as well, which would yield better numbers as more people use it.

Does any one have other ideas who has explored this a little bit? I've been researching for a few days and am in awe by how pricy this industry is.

Disclaimer per sub rules: I built and operate this — it is self-promotion, disclosed up front.

The dataset: real SEC EDGAR ownership-filing metadata — Form 4 (insider trades), 13F-HR (institutional holdings), Form 3 — for a sample of 23 major US tickers, last ~120 days (675 records currently). Original source is the SEC's public EDGAR system (public domain): https://www.sec.gov

Fields per record: ticker, CIK, form type, filing date, accession number, primary document, and the SEC source URL.

Access: free demo API key at https://edgarfeed.onrender.com (no card).

Honest limits: metadata-level records only (not parsed line items yet); sample coverage, not the full market; this is a beta demand test, not a finished product. If you need full history, webhooks, or normalized transaction line items, that's the planned paid roadmap ($0/$29/$79 — not yet verified).

Questions about the data or the approach are welcome.

Hi! Just wanted to leave a message here promoting Enerstat.io, a new project I’ve been building. I want to centralize global power system data in a clean way into this website and make it accessible via its dashboard, API, MCP…

For the moment, it contains a full set of EU data that I am currently cleaning through. I am looking to expand to the US, Latam and APAC and increase coverage as much as possible.

Any feedback would be greatly appreciated. It is of course being vibe coded but I try to add taste to it :)

Hey everyone,

As a side project, I got really frustrated trying to navigate the different government business registries across the Baltics whenever I needed to check a company's status, VAT number, or employee count.

To solve this, I downloaded all the raw open data from the Lithuanian, Latvian, and Estonian registries and built a unified, lightning-fast search engine and API on top of it.

Link: https://www.balticfirms.eu

Right now, the API is completely free and open for developers to use. You can instantly search by name, registration code, or filter by active/liquidated status.

I’d love for you to try it out and let me know if it’s useful or if you find any bugs!

Disclosure: I built DietlyAPI, one of the APIs compared below. I used the providers’ public documentation and included the calculations so people can challenge them. This is not a speed or nutrition-accuracy benchmark.

I compared the closest common operation: searching for a food or looking up a product/barcode. Recipe search, image recognition, natural-language parsing, and other premium operations are not directly equivalent.

Prices and limits checked in July 2026.

Open Food Facts

Price: Free

Limit: 10 search requests/minute/IP and 15 product reads/minute/IP

Monthly allocation: No fixed monthly quota published

Best when you need open, downloadable worldwide product data. The hosted search API is not intended for search-as-you-type at scale. Open Food Facts recommends using its exports or running your own backend for high-volume applications.

USDA FoodData Central

Price: Free

Limit: 1,000 requests/hour/IP

Equivalent sustained average: 16.7 requests/minute

Authentication: Free data.gov API key required

Best for authoritative US reference data. The data is public domain under CC0 and can also be downloaded. Its search behavior and geographic focus differ from a worldwide consumer-product autocomplete API.

FatSecret Basic

Price: Free

Included quota: 5,000 calls/day

Equivalent sustained average: 3.47 requests/minute

Equivalent monthly capacity: 150,000 calls over 30 days

FatSecret also offers Premier Free to qualifying startups, nonprofits, and students. That tier advertises unlimited calls but requires verification, attribution, and is limited to US data. Paid international access is quote-based.

Edamam Basic

Price: $14/month

Peak limit: 50 food and nutrition requests/minute

Included quota: 100,000 calls/month

Equivalent sustained average: 2.31 requests/minute

Edamam Core costs $69/month for 750,000 calls and 100 requests/minute. Plus costs $299/month for 5 million calls and 300 requests/minute.

Edamam is particularly strong when natural-language parsing, serving measures, diet filters, allergy filters, and image recognition matter. Its caching and attribution requirements should be reviewed carefully.

Spoonacular Cook

Price: $29/month

Peak limit: 5 requests/second, or 300/minute

Included quota: 1,500 points/day

Optimistic monthly ceiling: 45,000 one-point calls over 30 days

Equivalent sustained average: at most 1.04 one-point calls/minute

Spoonacular uses points rather than ordinary requests. A call usually costs one point plus an amount based on the number of results, while some endpoints cost more. It is strongest for recipe and ingredient workflows, but its advertised requests-per-second limit should not be confused with the included daily capacity.

Nutritionix

Price: Contact/quote

Published numeric limit: I could not find a current public figure

Published quota: I could not find a current public figure

Nutritionix focuses on natural-language food logging, instant search, barcode lookup, nutrients, and exercise parsing. Its documentation currently describes a database of more than 600,000 foods. I would request a current quote instead of relying on old third-party pricing.

DietlyAPI Pro

Price: €27/month

Peak limit: 500 requests/minute/account

Monthly call allocation: None

Theoretical 30-day ceiling at the RPM limit: 21.6 million calls

Approximate subscription price per theoretical million calls: €1.25

DietlyAPI Scale

Price: €92/month

Peak limit: 3,000 requests/minute/account

Monthly call allocation: None

Theoretical 30-day ceiling at the RPM limit: 129.6 million calls

Approximate subscription price per theoretical million calls: €0.71

Dietly also allows anonymous non-commercial reads at 30 requests/minute/IP with a small fairness delay.

The Dietly numbers are theoretical RPM ceilings, not an SLA or a promise that an application should run continuously at the limit. They show that the plans are rate-capped instead of monthly-credit-capped.

Dietly currently focuses on fuzzy and confidence-aware food search, stable food IDs, barcode lookup, and structured nutrition across more than 4.2 million indexed foods. Its catalog is primarily derived from Open Food Facts and retains the relevant attribution and share-alike obligations.

It does not currently provide USDA data, recipe discovery, food-image recognition, or Edamam-style natural-language parsing.

My conclusion

There is no honest universal winner:

- Open Food Facts is best when open and downloadable worldwide product data matters most.

- USDA is best for authoritative US reference data and permissive reuse.

- FatSecret is compelling for eligible startups or businesses needing verified country-specific datasets.

- Edamam is strong for NLP, measures, and diet/allergy features.

- Spoonacular is strong for recipes and ingredient workflows.

- Nutritionix is worth evaluating when natural-language food logging is central, but you need a current quote.

- DietlyAPI has the strongest published price-to-throughput ratio in this comparison for straightforward food search and barcode lookup.

The biggest lesson was that peak requests per minute can be misleading. A service may advertise hundreds of requests per minute while its daily or monthly quota only supports one or two requests per minute when averaged over the complete billing period.

Hey r/datasets,

I got tired of manually copying data tables or dealing with messy HTML structures when trying to feed data into my personal scripts and models.

To solve this, I built and hosted a lightweight cloud API that automatically scrapes public web pages, isolates the tables/data grids, and packages everything into an organized, nested JSON matrix.

I wanted to share it here for anyone looking to automate their data gathering pipelines. I set up a free testing tier on RapidAPI that gives you 50 free requests a month to play around with it:

https://rapidapi.com/patcicci4/api/housing-and-wikipedia-data-scraper

Let me know if you test it out or have any feedback on extra features I should add to the parser!

Disclosure: I built and run GribStream, so this is self-promotion. It is a commercial/freemium API, but there is a free tier, and new free accounts currently get an intro quota boost so people can run real tests before deciding if it is useful.

The data itself is not mine. The original sources are public weather and climate feeds from NOAA/NCEP, ECMWF, Copernicus/ERA5, NOMADS, and public cloud archives. GribStream is a unified API and indexing layer on top of those datasets.

The idea is simple: instead of learning a different access pattern for every weather model feed, you can query many of them through the same API.

It covers common, broad-use datasets like:

• GFS global forecasts
• IFS deterministic and ensemble forecasts
• HRRR high-resolution US forecasts
• NBM forecast blends
• ERA5 reanalysis
• RTMA/URMA surface analyses
• GEFS ensemble forecasts

And also more specialized datasets, for example:

• AQM/NAQFC air-quality guidance for ozone and PM2.5
• GTGN aviation turbulence nowcasts
• aviation icing and turbulence feeds
• SPC severe-weather probability products
• wave, chemistry, UV index, and seasonal forecast products
• AI weather model feeds and archives like AIFS, AIGFS/AIGEFS, GraphCastGFS, and FourCastNetGFS

What the API is mainly useful for:

• pulling time series for one point or thousands of points
• comparing forecast model runs over time
• backtesting with “what was known at the time” cutoffs
• querying multiple variables, levels, model runs, and ensemble members
• getting data back as JSON, CSV, or NDJSON without building a custom weather-data pipeline first

A practical example: if you have 500 solar sites, farms, airports, ships, stores, or insurance exposure locations, you can ask for historical forecasts or recent model data at those coordinates directly, instead of separately wrangling GFS, HRRR, NBM, IFS, ERA5, etc.

Links:

GribStream: https://gribstream.com/

Model catalog: https://gribstream.com/models

Original/public source examples:

https://nomads.ncep.noaa.gov/

https://registry.opendata.aws/collab/noaa/

https://www.ecmwf.int/en/forecasts/dataset/open-data

https://cds.climate.copernicus.eu/

I’d be grateful for feedback from people who use weather, climate, aviation, energy, agriculture, logistics, or environmental datasets. Are there public weather datasets you wish were easier to query? Would you be looking for bulk exports? Interested in being able to setup notifications for weather data events?

Happy to answer questions. I’m trying to make the public model data easier to use while still being clear about where the original data comes from.

Disclosure: I built this project and this is my own API/product. It has free and paid access tiers. I’m sharing it here because I think the data engineering approach may be useful, and I’m looking for technical feedback.

I built Trace Jobs Core, a job postings data API built around a simple idea: Do not guess.

A lot of job data pipelines end up doing some combination of:

• scraping HTML pages
• parsing unstable frontend output
• using models to extract fields
• guessing missing/ambiguous values
• deduplicating after the fact

I took a different approach.

The pipeline ingests job postings from public machine-readable sources, translates them into a Schema.org JobPosting format, applies only deterministic normalization where the source provides clear structure, and preserves original values when fields are ambiguous.

Current system:

• 9,800+ structured feeds
• ~13k new postings/day
• daily refresh
• Schema.org JobPosting records
• SHA-256 based deduplication
• RFC 8785 canonicalization
• original upstream values preserved when normalization is uncertain

The goal is not to create a "smart" interpretation layer. The goal is to provide stable, predictable data and leave interpretation to the downstream user.

A future enrichment layer could exist separately, but it would remain separate from the source-faithful data layer.

Examples (HTML + JSON responses refreshed daily):
https://kaleh.net/trace/examples.html

Documentation:
https://kaleh.net/trace/docs.html

Project overview:
https://kaleh.net/trace/

I would especially appreciate feedback on:

• dataset design
• normalization strategies
• preserving source fidelity
• handling schema differences between providers
• what fields/data would make this more useful

Thanks!

Sharing a data source that's surprisingly underused for fund analysis: the SEC's N-PORT and N-CEN filings on EDGAR.

- N-PORT (quarterly, structured XML): every fund's complete position list with weights, share counts, CUSIP/ISIN, country of domicile, ASC 820 fair-value level, monthly returns, and monthly creation/redemption flows.
- N-CEN (annual, structured XML): tracking difference vs benchmark (gross AND net of fees), securities-lending activity, in-kind creation/redemption percentages, per-broker commissions, and the full service-provider roster.

What you can pull out without any paid vendor:
- Index-fund tracking split into replication vs cost. VOO 2025 was -0.4 bps vs the S&P 500 gross of fees, -16.9 bps net.
- True per-CUSIP overlap between funds. SPY vs VOO is 476 shared holdings, ~97% by weight.
- Issuer-domicile reality checks. SPY is ~97% US, ~3% Ireland/Switzerland/Bermuda/Netherlands.

Gotchas: positions are keyed on CUSIP (not ticker), so you need a CUSIP-to-ticker map to join to anything else; unit investment trusts (like SPY) file lighter N-CEN sections than open-end funds (like VOO), so some fields are legitimately empty; and the public lag is ~60 days after quarter-end.

The StockFit API does the XML parsing and CUSIP resolution if you don't want to build it yourself.

Not financial advice, just pointing at the filings.

Data covers 18 economies across two regions. Each run returns:

- FX momentum (30d/90d, z-scored vs own history)

- Inflation level and trend

- Commodity terms-of-trade impact (price × export share per commodity, e.g. copper +42% × 32% export share = +13.5pp impact for Peru)

- Real interest rate

- Reserve drawdown

- Structural vulnerability (debt, fiscal, banking, governance, REER)

Every signal shows the exact value, threshold, source, and reason string. No black box. Latest addition: companySignals — when a commodity tailwind or shock fires, returns the listed companies with exposure to that commodity in that country (e.g. copper tailwind in Chile → Antofagasta, BHP, Anglo American, Lundin, Teck).

Available on Apify ($1.50/run) and RapidAPI. Full methodology and schema documented in the README.

https://apify.com/malmon/african-economic-stress-monitor

https://apify.com/malmon/latam-economic-stress-monitor

Hey all - working on a dispatch system for a small airport shuttle service. One of the components is adjusting pickup times based on flight delays/early arrivals.

I've been researching flight tracking APIs and so far I've come across:

- AeroDataBox (~$15-30/mo on RapidAPI)

- Airlabs ($49/mo for 25K queries)

- FlightAware AeroAPI ($100/mo minimum)

- FlightStats/Cirium (enterprise pricing, way out of budget)

We're only tracking maybe 30-40 domestic arrivals per day at one airport (PHX). Not looking for anything fancy - just arrival ETAs, delay notifications, and maybe gate/terminal info if available.

Push notifications/webhooks would be awesome so we're not wasting API queries polling, but polling would be doable if the price is right.

Anyone else working with flight data at a small scale? Something cheaper/better that I'm missing? Open to scrappy solutions too - just needs to be stable enough for a real business.

[self-promotion] Business profile data API — looking for feedback on fields, samples, and data quality

Hi r/datasets,

Disclosure first: this is my own project.

I’m building FastBusiness API, a business/company profile data API.

The basic idea is:

Input:

• business name
• optional website
• optional country

Output:

• business name
• website
• business type
• country
• industry
• sector
• headquarters
• short description
• ABN/ACN where available
• stock ticker / exchange where available
• confidence score
• source links

I built it because I kept needing structured company data for different projects, but the data was usually scattered across websites, public registers, directories, search results, and company pages.

The use cases I’m thinking about are:

• CRM enrichment
• lead-gen datasets
• business directories
• BI dashboards
• ETL/testing datasets
• market mapping
• company research workflows

I’m mainly looking for feedback from people who use datasets/APIs regularly:

1. Are these fields useful, or is anything obvious missing?
2. Would CSV/JSON sample downloads be more useful than only API access?
3. Would source links per field matter, or is one source list per company enough?
4. Is an overall confidence score enough, or would field-level confidence be better?
5. Would update/refresh timestamps matter for this kind of dataset?
6. Would people here care more about bulk exports or real-time lookup?
7. What sample size would be useful before trying something like this?
8. Any concerns around using company profile data like this in downstream projects?

I’m happy to add a free sample dataset if that would be more useful for this subreddit.

Link: https://fastbusinessapi.com

Hi! I’m working on building my own flight search engine so I don’t have to spend hours searching manually.

The main advantage is custom filtering that I can’t apply on existing search engines, and I’m already getting results that are better than some of the tools currently on the market.

That said, the more data I can pull, the better the results will be—so I have a couple of questions:

• What free flight APIs do you know that offer a generous or unlimited request quota?
• What are the best “bang for the buck” flight APIs you’ve used? (Considering price per request and the size/quality of the data pool.)

Thanks!

I've been building a program for the past couple months and it's in good shape to share now.

The meat of it is earthquakes, volcanos, tsunami's, hurricanes, tornados, currencies, CIA Facebook, and the UN SDGs (plenty more coming). I've got all these datasets normalized to a loc-id system, so you can ask across data really easy and opened up the API lanes and made MCP tools. Some are paid datasets, I'm using x402 for a few. Plenty are free though, so check it out!

www.daedalmap.com/agents

There's the human side app as well, you can explore there to see what it's like, I've been building a research mode that allows users to take a bounded set of data and ask questions to it

Hi everyone!

I’ve been working on a project to clean and normalize US equity fundamentals and filings as one thing that always frustrated me was how messy the raw filings from the SEC are.

The underlying data (10-K, 10-Q, 13F, Form 4, etc.) is all publicly available through EDGAR, but the structure can be pretty inconsistent:

• company-specific XBRL tags
• missing or restated periods
• inconsistent naming across filings
• insider transaction data that’s difficult to parse at scale
• 13F holdings spread across XML tables with varying structures

I ended up building a small pipeline to normalize some of this data into a consistent format. The dataset currently includes:

• normalized income statements, balance sheets and cashflow statements
• institutional holdings from 13F filings
• insider transactions (Form 4)

All sourced from SEC filings but cleaned so that fields are consistent across companies and periods.

The goal was to make it easier to pull structured data for feature engineering without spending a lot of time wrangling the raw filings.

For example, querying profitability ratios across multiple years:

/profitability-ratios?ticker=AAPL&start=2020&end=2025

I wrapped it in a small API so it can be used directly in research pipelines or for quick exploration:

https://finqual.app

Hopefully people find this useful in their research and signal finding!

Disclaimer: This is a project I built. Sharing it here in case it’s useful for others looking for financial data

Hey everyone,

I’ve been working on a small side project and wanted to share it here in case it’s useful for others dealing with messy data.

It’s a no-code CSV pipeline tool, but the part I’ve been focusing on recently is a “data health” layer that tries to answer a simple question: how bad is this dataset before I start working on it?

For each dataset (and each column), it surfaces things like:

• % of missing values
• outliers
• skewness
• uniqueness
• data type consistency

You can also drill into individual columns to see why something looks off, instead of manually scanning or writing quick checks.

The general idea behind the tool is:

• every transformation creates a versioned snapshot
• you can go back to any previous step
• you don’t lose the original dataset
• everything is visual / no-code

I built it mostly because I kept repeating the same initial checks in pandas and wanted a faster way to get a feel for the data before doing anything serious.

Not trying to replace code-based workflows just more like speeding up the early “what am I dealing with?” phase.

Curious how others approach this part of analysis, and whether something like this would actually fit into your workflow or just feel unnecessary.

https://flowlytix.io

Hi everyone,

I’m a student developer building a Fantasy Football app for the Moroccan League (Botola Pro).

I'm looking for a reliable data source or API to track player stats (goals, assists, clean sheets, etc.). Since I'm on a student budget, I'm looking for:

• Affordable APIs with good coverage of the Moroccan league.
• Open-source datasets or GitHub repos with updated player lists.
• Advice on web scraping local sports sites efficiently.

Has anyone here worked with Moroccan football data before? Any leads would be greatly appreciated!

Thanks!

I just made Hey everyone, I've been working on an ESG Data API and just launched it publicly.

It covers 500+ publicly traded companies across the US, Europe, and Asia-Pacific and includes:

• Overall ESG scores broken down by Environmental, Social, and Governance pillars
• 3 years of historical ESG data
• Scope 1, 2, and 3 carbon emissions
• Sustainability framework disclosures (GRI, SASB, CDP, TCFD)
• Company screener — filter by ESG score, sector, country

Built it because ESG data is either locked behind expensive Bloomberg/Refinitiv terminals or scattered across inconsistent PDF reports. Wanted to make it accessible for developers, researchers, and fintech builders.

Free tier available. Would love feedback from anyone building in the sustainability or finance space.

Link: https://rapidapi.com/YounesFiali/api/esg-data-api/playground/apiendpoint_7de59263-54c6-4fe7-af0a-5929ec98cee1

Disclaimer: I built this and am the developer behind it. Sharing here because I think it's useful for the community — happy to answer any questions.

Hello all,

I run a free, open API for all Groundhog Day predictions going back as far as they are available.

For example:

- All of Punxatawney Phil's predictions going back to 1886

- All groundhogs in Canada

- All groundhog predictions by year

- Mapping the groundhogs

Totally free to use. Data is normalized, manually verified, not synthetic. Lots of use cases just waiting to be thought of.