Hacker News

points by bijowo1676 3 days ago | 0 comments

if you put stuff available to download, dont be surprised that someone will download it

What is the actual news here, can anyone explain ????

They didn't do it in the intended manner.

> The crawler did not respect our robots.txt, (which it did retrieve), ignored our Terms of Service, (which forbid scraping), and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only.

[–] Sopel 2 days ago

> did not respect our robots.txt

I don't see a problem

> ignored our Terms of Service

ban them from the service then and case closed?

> and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only

because it's been proven they have been used commercially?

[–] trollbridge 2 days ago

I do. It's wrong to ignore a robots.txt.

The fact someone else may have licenced content for commercial use doesn't entitle someone else to do the same without paying for it. That's the whole point of copyright law.

[–] stickfigure 2 days ago

> It's wrong to ignore a robots.txt.

It's not nice. It is, however, not illegal.

And before you agitate against that, think deeply about the consequences. A world where website owners can legally dictate exactly what software you use to access their content.

[–] BizarroLand 2 days ago

They didn't say illegal, they said wrong. And they were right. You even agree with them by saying "It's not nice". Non-nice things are wrong in some way.

And, we already live in a world where website owners can legally dictate exactly what software you use to access their content. They can require that you log in with specific browsers running specific software on your computer, they way online schools run exam proxies.

[–] stickfigure 2 days ago

We do not in fact live in that world. It is not illegal to violate the terms of service of a website.

A website can block Xyz user agent, put Xyz in robots.txt, or even put "No using Xyz!" in a popup and you cannot be prosecuted for using Xyz. It might be hard for you to evade their requirements, but it's not illegal.

Whether or not it's wrong is a matter of opinion. Yours differs from mine, that's fine.

[–] trollbridge 2 days ago

It is illegal to download copyrighted material from a website and make copies of it in violation of its licence.

I can’t go scrape New York Times articles and make copies of them without first licensing them from NYT.

[–] BizarroLand 2 days ago

Wrong != Illegal

It is wrong to pretend you are deaf to get out of an awkward conversation. No one is going to put cuffs on you for that.

[–] trollbridge 2 days ago

The copyright terms on my own site make ignoring it and scraping it expressly illegal.

Unless copyright law has been entirely repealed?

[–] greycol 2 days ago

US case law may not be on your side there. Generally unless you require a login to access the site (and even then you can only really ban for terms of service rather than recover damages unless they hacked around login requirements) judges have found that using scrappers on publicly served data is fair use. It gets murkier with creative content (rather than straight data) in what they can do with it after scrapping but the scrapping itself would still be 'legal' in the they can do it without judicial consequence sense.

It's one of the reasons why sites that served a lot of data used to have APIs or back channels to get that data even if they technically made money by serving ads on pages of that data, because they were going to get scrapped no matter what, so having a legitimate way to get data that behaved nicely with the servers and was nicely readable by requestors was win-win or at least not lose-lose.

[–] tarsierscripter 2 days ago

The news is that people download it so much that they saturate your server’s connection and/or peg its CPU or disk (whichever is the weakest link in the chain). We used to call this a Denial of Service attack but now it’s apparently just a cost of doing business, even for people who just want to run a non-commercial hobby site.

[–] bijowo1676 2 days ago

its not a denial of service because there was never an intent to deny the service

[–] tarsierscripter 2 days ago

The effect is a denial of service. Service is denied because the server is overloaded. The intent is at best callous indifference and at worst is copyright infringement.

It’s comparable to broadcasting radio interference as a side effect of some industrial process. If a company were to do that they’d be slapped down hard by the FCC. The unfortunate reality we live in is that we have no equivalent regulation on the internet. Thus the situation has become a tragedy of the commons: small websites suffer denial of service due to unsustainable and unwanted bot traffic.

[–] 233mhz 2 days ago

https://therecord.media/linkedin-wins-court-order-blocking-m...

Which is it then? Megacorps can enforce their TOS through the law but small companies cannot?

[–] well_ackshually 3 days ago

Man if only there was such a thing as good manners, good behaviour, accepted practices from a community to not massively hammer servers. If only these websites had been up for a decade with very few problems, before a notoriously manipulative, lying, deceiving, abusive company that makes money off of child abusers came in and just went hurr durr free data in the WORST possible way instead of just politely asking for a dump.

[–] bijowo1676 2 days ago

Internet is untrusted network, there are a lot of malicious and gray actors. There is no community here.

it is in fact engineering incompetence and loose engineering, if you cant handle a burst of traffic

[–] numpad0 3 days ago

"Leopard ate MY face" ?