Ethics and web scraping – How do you handle it?

Web scraping often raises questions about what is and isn’t allowed. But beyond what’s technically or legally possible, we always ask ourselves: does it feel right? At Scrape IT, we take a conscious and careful approach to these questions.

Where do you draw the line?

There are many ethical dilemmas in web scraping that require careful thought:

• Is it allowed?

• Do we feel okay about it?

• What will the client do with the data?

Because these questions often don’t have clear-cut answers, we regularly vote within the Scrape IT team on whether to take on a project. If even one of us feels uncomfortable with it, we don’t do it. As a result, we sometimes turn down assignments.

A real-life example: Housing data

One of the things we scrape is housing data. Project developers use this data to identify promising areas for new construction. That’s fine by us — if our data helps make better decisions about where to build, we’re happy to contribute.

But the same data can also be used for less welcome purposes. We sometimes get requests to detect when people move house, since we can see when a property is sold. Certain businesses want to act on that signal right away and send marketing materials.

But let’s be honest: no one enjoys entering their new home and immediately tripping over a pile of flyers from companies trying to sell them stuff. In our view, this is a scattergun approach — most people aren’t interested, it wastes paper, harms the environment, and annoys people who are already dealing with the stress of moving. So we say no to those kinds of requests.

How we make decisions

Sometimes companies ask us for a copy of our ‘ethics policy’ so they can reuse it. But that’s tricky — our ethical compass is based on gut feeling, not just rules.

Still, we do ask ourselves a few key questions every time:

Does it feel right?
Does it truly help the client?
Does it avoid harm to others or the world?
Does it contribute to something positive?
Do we treat personal data with care?

It’s the combination of these questions that helps us decide. We discuss it together and make a group decision. Ultimately, our ethical compass is intuitive — and that’s okay.

Ethical choices require courage

Ethics isn’t always objective or measurable — but it’s still essential. Especially in a field as powerful as web scraping. We believe that listening to your gut now and then is not just fine — it’s necessary.

Got a question or want to spar with us about data? We’re happy to think along – honestly and transparently.