Hacker News
points by Sopel 2 days ago | 0 comments
> did not respect our robots.txt

I don't see a problem

> ignored our Terms of Service

ban them from the service then and case closed?

> and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only

because it's been proven they have been used commercially?

[–] trollbridge 2 days ago
I do. It's wrong to ignore a robots.txt.

The fact someone else may have licenced content for commercial use doesn't entitle someone else to do the same without paying for it. That's the whole point of copyright law.

[–] stickfigure 2 days ago
> It's wrong to ignore a robots.txt.

It's not nice. It is, however, not illegal.

And before you agitate against that, think deeply about the consequences. A world where website owners can legally dictate exactly what software you use to access their content.

[–] BizarroLand 2 days ago
They didn't say illegal, they said wrong. And they were right. You even agree with them by saying "It's not nice". Non-nice things are wrong in some way.

And, we already live in a world where website owners can legally dictate exactly what software you use to access their content. They can require that you log in with specific browsers running specific software on your computer, they way online schools run exam proxies.

[–] stickfigure 2 days ago
We do not in fact live in that world. It is not illegal to violate the terms of service of a website.

A website can block Xyz user agent, put Xyz in robots.txt, or even put "No using Xyz!" in a popup and you cannot be prosecuted for using Xyz. It might be hard for you to evade their requirements, but it's not illegal.

Whether or not it's wrong is a matter of opinion. Yours differs from mine, that's fine.

[–] trollbridge 2 days ago
It is illegal to download copyrighted material from a website and make copies of it in violation of its licence.

I can’t go scrape New York Times articles and make copies of them without first licensing them from NYT.

[–] BizarroLand 2 days ago
Wrong != Illegal

It is wrong to pretend you are deaf to get out of an awkward conversation. No one is going to put cuffs on you for that.

[–] stickfigure 2 days ago
Okay then, how about this:

Ignoring robots.txt is not wrong. That's my opinion, just as valid as yours.

It also happens that the legal system agrees with my opinion of the wrongness of this act.

Also, you did say this:

> where website owners can legally dictate

Which is, as I pointed out, false.

[–] trollbridge 2 days ago
The copyright terms on my own site make ignoring it and scraping it expressly illegal.

Unless copyright law has been entirely repealed?

[–] greycol 2 days ago
US case law may not be on your side there. Generally unless you require a login to access the site (and even then you can only really ban for terms of service rather than recover damages unless they hacked around login requirements) judges have found that using scrappers on publicly served data is fair use. It gets murkier with creative content (rather than straight data) in what they can do with it after scrapping but the scrapping itself would still be 'legal' in the they can do it without judicial consequence sense.

It's one of the reasons why sites that served a lot of data used to have APIs or back channels to get that data even if they technically made money by serving ads on pages of that data, because they were going to get scrapped no matter what, so having a legitimate way to get data that behaved nicely with the servers and was nicely readable by requestors was win-win or at least not lose-lose.