I participated in a very unsuccessful group-buying event. The organizer was very serious and found a wholesale platform that claimed to have "everything," but it turned out that half of the items our group wanted were not available on that platform, and the other half were available but required separate negotiation. In the end, the organizer visited four stores himself, using the platform account only to check prices from beginning to end.
When evaluating data aggregation platforms, I kept thinking about that group-buying deal. To continuously scrape data from many external sources into my own storage, the first question to ask was, "Use an existing platform or write your own?" In the open-source world, the two most frequently recommended answers are Airbyte and Meltano. I carefully evaluated them and didn't install either. This article discusses what each is, when they're worthwhile, and what they taught me.
Airbyte: An ELT platform with interfaces
Airbyte is a suite of persistent services: a web interface, a scheduler, and workers, each running in its own independent connector container. The core program is licensed ELv2, permitted for internal use; the connectors themselves are mostly MIT (Made in Taiwan). Its strength lies in its catalog: hundreds of SaaS, database, and archive systems as sources, which can be easily configured for synchronization once installed, and data begins flowing with a few clicks on a webpage. For a team looking to centralize data from a dozen or so SaaS systems within a company, this is a godsend.
The weakness is deployment; it essentially requires Docker or Kubernetes. On Windows, it requires Docker Desktop, which means enabling Hyper-V. Moreover, it's "a set of services," not "a program," meaning you're essentially running a small data department on your own machine.
Meltano: Python command line
Meltano is an MIT-licensed command-line tool that revolves around the Singer specification: the source is called tap, the destination is called target, and a standardized JSON stream runs in between. tap is responsible for outputting data from a location, and target is responsible for inputting data into a location; the two sides don't understand each other and communicate through the intermediate format. It can run natively in Python, doesn't require a container, and its project structure is a YAML file plus a few plugins, making it much lighter than Airbyte.
The weakness also lies in the directory: the quality of Singer taps varies greatly. Some are great, some haven't been updated in three years, and there are mostly no obscure sources, or if there are, you'll find that the last submission was a graduation project by someone who has already graduated.
Coverage determines everything
My list of sources includes over a hundred groups, mostly individual organizations' own REST APIs, a few WebSocket streams, a few SSE endpoints, and SDMX commonly used by statistical agencies. There are almost no readily available connectors for these in either of the directories.
In other words, even after installing the platform, I still have to write connectors for the vast majority of sources myself, just using their specifications and maintaining an additional set of services. This calculation is quick to complete, much like that group-buying experience: the platform has things I don't need, and the platform doesn't have what I need, so in the end I still have to go to the stores myself.
Conversely, if your source is common SaaS and databases, these two tools save you the maintenance of an entire batch of connectors, which is a significant expense, so use them when necessary. Don't think you're cool just because you read this article; writing your own content is never cool, it's just that sometimes you have no other choice.
The ten questions they left me with
Rejecting a tool doesn't mean its experience is worthless. Airbyte and Meltano exist because people who write their own connectors repeatedly encounter the same set of problems: timeouts, rate limiting, scepter expiration, source schema changes, retries, checkpoints, incremental fetching, deduplication, scheduling, and monitoring.
When I first saw this list, I felt a little uneasy because about six of the items were things I "planned to deal with later." "Deal with later" basically means "waiting for things to go wrong." So I copied down these ten items as an acceptance checklist for my framework, making sure to clearly explain how each of these ten things would be handled before each source module went live. Some answers might be "this source doesn't need it," but that's still an answer; the important thing is that I've thought about it.
At the heart of our custom framework is a login table, one entry per source: real-time category, crawling method, authentication settings, polling strategy, dataset list, and status. There are only five statuses: Planning, Under Development, Launched, Paused, and Retired. Five are enough; any more and you'll start inventing things like "semi-launched," and nobody will know what that means.
Mini-drama: Isn't it tiring to write it yourself?
"You're going to write all 100+ sources yourself? Won't you start to question your existence?"
"Yes, around the thirtieth one."
"So how did you manage to get through it?"
"Because the first twenty-nine elements helped the framework take shape. After the thirtieth element, each one is just filling out a form, specifying the timeout, speed limit, and pagination."
"So the difficult part is at the beginning."
"The difficult part is the beginning, and there are no tools to help you with that part because it involves figuring out what your own data looks like."
"This sentence sounds like some kind of life philosophy."
"Every sentence in data engineering eventually becomes a life lesson, which is an occupational hazard."
Several concepts borrowed in the same round
I didn't install Dagster or dbt. Instead, I borrowed dbt's SQL model directory structure and layered data repository concept, using DuckDB and Parquet as my local machine. Some reference projects use Streamlit as their interface, but I didn't use their interface; I only ported its anomaly detection formulas. I didn't use any cloud services; vectorization was handled by my local Ollama.
"Borrowing concepts without installing tools" is something I've done frequently in recent years. Tools become outdated, depend on others, and require updates; concepts don't. Understanding why a tool is designed the way it is is more valuable than installing it, even though installing it might feel more rewarding.
My judgment criteria
The value of an off-the-shelf platform is directly proportional to how much of your source material falls into its directory. If most of your source material falls into the directory, use the platform; don't overextend yourself. If most of your source material falls outside the directory, the platform will only offer you its specifications and deployment methods. In that case, you might as well write your own and honestly admit that you are redoing ten things they have already done.
What happened after that group-buying event? After visiting four stores, the organizer compiled a table listing "what each store has," which she would use for the next group-buying event. She said that table was better than any platform's because it originated from her own needs. I think she probably doesn't know that what she did has a name: "login table."
The origins of the two tools
Meltano started as an internal project at GitLab, began in 2018, and spun off into a separate company in 2021. It's built on the Singer specification, an open standard proposed in 2017 by a company called Stitch. Stitch defines a JSON format for "how sources send data and how destinations receive data," allowing any tap to connect to any target. The idea is brilliant, but the practical problem is that "anyone writes it" means no one is responsible for quality control.
Airbyte was founded in 2020, much later, but it has always taken a "platform" approach: it has interfaces, scheduling, and a connector marketplace; you don't need to write YAML, just click around. It has grown rapidly, with its connector catalog reaching hundreds within a few years. Its core license uses ELv2, which allows for personal use, modification, and even commercial use, but prohibits using it to create a competing managed service. This type of "open-source but cloud service provider-protected" license has become increasingly common in recent years, which is another topic altogether.
Both are talking about something called ELT: first, the data is transferred as is (Extract, Load), and then transformed at the destination (Transform). Unlike the old ETL, which transforms the data before transferring it, ELT's logic is that storage is cheaper and the destination has stronger processing power, so we transfer the data first, and transformation can be done later. My self-written framework follows this idea: the raw response is archived first, then normalized before being added to the database.
Three common misunderstandings
The first misconception is that "if it's in the connector directory, it means it works." Just because it's in the directory doesn't mean it supports the specific API version, authentication method, or pagination mode you need. Checking the last update date and issue list is more important than seeing it in the directory.
The second misconception is that "the platform will handle schema changes for you." It will detect the change, stop, and ask you what to do. The decision is still yours. This becomes a daily routine when you have a hundred sources, not much different from using a framework you wrote yourself.
The third misconception is that "writing your own framework is always worse." A self-written framework only does what you need, without unnecessary complexity. It will certainly lose in terms of versatility, but you don't need versatility; what you need is a stable supply from those hundred sources. The value of a tool should be measured by the amount of demand you have, not by the number of features it has.
A small exercise you can try over the weekend
List your five most important data sources—anything goes: weather, exchange rates, open data from a government, RSS feeds from a website you frequently visit. Then check Airbyte's connector directory and Singer's tap list, noting how many have readily available data, how many have been updated in a long time, and how many don't.
If four out of the five have readily available and maintained options, you should use the platform and not try to write your own after reading this. If four out of the five don't, what you just did is the same assessment I did, only my list is a bit longer. The focus of this exercise isn't the conclusion, but that you'll learn what a "connector catalog" actually looks like and what "still under maintenance" means during the research process—things that will be very useful when you actually need to make a decision.
Postscript
The organizer later held several more group-buying events, each time going even more smoothly. Her spreadsheet grew from "which store has what" to "which store has what, delivery time, who is responsible for tracking, and what problems occurred last time." Looking at that spreadsheet, I thought, isn't this just a complete data pipeline log sheet, only she used Google Sheets? The tool isn't important; what's important is thinking clearly about the problem and writing it down. I guess she'll never read this article, but if she does, I want to tell her: Your spreadsheet is very professional.
Official website: airbyte.com , meltano.com