Question 1

Is this legal?

Accepted Answer

It depends on what you are collecting and from where, and it is worth settling before anyone writes code. I work with publicly accessible data, respect robots rules and rate limits, and keep collection paced so it does not burden the source. What I will not build is anything that needs someone else's login, bypasses authentication, or defeats bot protection to get at data that is not public. If your requirement depends on that, I am the wrong person, and I will say so on the first call rather than after you have paid.

Question 2

What happens when a site changes its layout?

Accepted Answer

It will, and the system is designed on that assumption. Extraction is validated against what it produces, so a selector that suddenly returns empty raises an alert instead of writing nothing quietly. Extraction logic is also separated per source, so fixing one is a small change rather than a rewrite. On an ongoing arrangement I handle those fixes. On a fixed price build I document how to make them and record a walkthrough.

Question 3

How do you handle getting blocked?

Accepted Answer

Mostly by not deserving it. Requests get paced per source, run through a queue rather than in parallel bursts, and back off when a source signals it wants less traffic. Politeness fixes most blocking, because most blocking is a response to volume. Where a source is behind serious bot protection, that is a case by case conversation and sometimes the honest answer is that the data is not reliably available.

Question 4

How fresh will the data be?

Accepted Answer

You set that, and it drives the cost. Every fifteen minutes across three sources is a different system from once a day across one. On the monitoring products I have built, alerts reached users within seconds of a match appearing, which meant collection had to run continuously rather than on a schedule. Tell me how fresh it needs to be and I will tell you what that costs to run.

Question 5

Where does the data live?

Accepted Answer

Usually PostgreSQL, because you will want to query it in ways you have not thought of yet and a relational schema handles that better than a pile of JSON files. Redis for the queue and for anything needing fast lookup. If you already have a warehouse you want it landing in, I write to that instead.

Question 6

Can you add this to a product we already run?

Accepted Answer

Yes. I read your codebase and work to its conventions rather than bolting on something that only I understand. The ingestion side usually sits well as a separate service with a defined interface, which keeps it from tangling with the rest of your application and makes it easier to replace later.

Question 7

What if an official API exists?

Accepted Answer

Then we use it, and I will check before quoting a scraper. An API is more stable, cheaper to run and less likely to break, so finding one is the best possible outcome for you even though it makes the project smaller. It happens more often than people expect, because the endpoint powering a site's own search is sometimes public and undocumented.

Project	Scale	Typical timeline	Notes
One off extract	A single source, a fixed dataset	3 to 7 days	You need the data once, cleaned and delivered as a file or a table. No scheduling, no monitoring.
Monitored pipeline	One to three sources, running continuously	2 to 5 weeks	Scheduled collection, retries, alerting when a source breaks, and storage designed for querying. The most common shape.
Monitoring product	Several sources, alerts to end users	4 to 8 weeks	Everything above plus user facing saved searches and notifications, which turns a pipeline into something you can charge for.
Ingestion for an existing platform	Many sources, one canonical model	Scoped case by case	Reconciling several sources that name the same thing differently, into one model where counts can be trusted.

Data pipelines*