Box Art
Welcome to RetroTracks
- Remember that awesome tune? We do too.
Home Search Music Systems Artists Forum
Forum

RetroTracks Forum - Looking to download everything


maddope
Posts: 2
Looking to download everything
« posted: Feb 8th 2016, 11:35 pm, last edited: Feb 9th 2016, 12:16 am »

Is there an easier way other than clicking every single link on every page???

I would like to archive the work here but it would take forever to have to click each and every download link

It could be made easier if extensions like DTA worked on the site but they dont because the download links don't show up as links at all - and even if you can get it to read them it downloads them with their Hash table name instead of the actual file name
Enverex
Posts: 343
(site admin)
Avatar
RE: Looking to download everything
« posted: Feb 9th 2016, 10:26 am »

In addition, it's also rather bad form to try and clone a website, especially without using any throttling and it'll generally result in you getting outright blocked from websites. Mainly because it puts a large amount of load on the server in question, for example your attempt to clone the site resulted in over 14000 requests (even excluding the music itself) at a rate of around 4 per second. Luckily that won't affect the site here as it's stupidly overspec'd anyway, but other sites aren't likely to be as lenient.
maddope
Posts: 2
RE: Looking to download everything
« posted: Feb 10th 2016, 12:57 am »

Thanks for the response - I understand all of that - ill just grab a few a day

thanks for replying
Enverex
Posts: 343
(site admin)
Avatar
RE: Looking to download everything
« posted: Feb 10th 2016, 6:33 pm »

You can grab as many as you like, as fast as you like if you're doing it by hand. The difference is if you try and use a tool or script, it normally does 10 or 100 a second, which is quite a jump in use!
Enverex
Posts: 343
(site admin)
Avatar
RE: Looking to download everything
« posted: Feb 9th 2016, 9:23 am »

Generally no, the site was designed mainly for listening on the site itself and for downloading the odd file (although downloading all tracks for a collection shouldn't take long unless it has an unusually large amount of music).

I had to resort to running the download links through a slightly odd mechanism because there was a large amount of bots that would just keep crawling through all the pages and repeatedly downloading tracks over and over.

The hash table name thing is actually purely for storage reasons rather than anything else, there's over 16000 tracks on the site right now so it needed to be future proof and using the track's actual name for storage is never a good idea for a handful of reasons.

So in short, not really, sorry. The site isn't designed for bulk downloading.
Pages
PREV
1
NEXT