EvilZone

Programming and Scripting => Scripting Languages => Topic started by: Kulverstukas on January 19, 2012, 03:44:04 PM

Title: [Python] Check links
Post by: Kulverstukas on January 19, 2012, 03:44:04 PM

Uhm this is a simple script I made to check a list of links for dead ones and pick out the good ones.
List is a text file with links. Usage is also simple: checkLinks.py filename
Good links will be output to a file named good_filename.

Script: checkLinks.py (http://newage.ql.lt/projects/python/checkLinks.py)

Title: Re: [Python] Check links
Post by: neusbeer on January 19, 2012, 04:22:35 PM

this one can be handy .. thnxs
does this also check file existence on sites?
if I have for example a large list with
http://<link>/pvt/serice.pwd
?

Title: Re: [Python] Check links
Post by: Kulverstukas on January 19, 2012, 05:50:30 PM

Sorry, no. It will only check if the link (or IP) is online and responsive. It does not check for file existence on remote hosts.

Title: Re: [Python] Check links
Post by: NeX on January 20, 2012, 05:25:31 PM

Quote from: neusbeer on January 19, 2012, 04:22:35 PM

this one can be handy .. thnxs
does this also check file existence on sites?
if I have for example a large list with
http://<link>/pvt/serice.pwd
?

wget --spider does the job :D
You can then combine it with -i for input file, i.e. bookmars file, or whatever, and -o for output..

Title: Re: [Python] Check links
Post by: neusbeer on January 20, 2012, 08:31:30 PM

yes I know.. but that's a slow method.. (which I'm using now)..

Title: Re: [Python] Check links
Post by: xzid on January 20, 2012, 09:40:54 PM

Quote from: neusbeer on January 19, 2012, 04:22:35 PM

this one can be handy .. thnxs
does this also check file existence on sites?
if I have for example a large list with
http://<link>/pvt/serice.pwd
?

in ruby

Code: [Select]

require 'net/http'
while x = gets
begin
    puts x if Net::HTTP.get_response(x.chomp, '/pvt/serice.pwd').is_a? Net::HTTPSuccess
rescue
end; end

run it:

Code: [Select]

~/code/rb $ ./httpcheck < hosts.txt > good.txt


Title: Re: [Python] Check links
Post by: neusbeer on January 22, 2012, 12:41:56 AM

Gonna test this one..
Ruby is rather new for me.  looks not to harshe.. thnxs

Title: Re: [Python] Check links
Post by: xzid on January 22, 2012, 02:32:12 AM

ruby is the lang I'm currently learning[~week], I already prefer it to python/perl. Seems you can hack stuff together very quickly which is exactly what I like(C is a different thing altogether)

... although in some ways it's unlike anything else I code, so I find it interesting.

Title: Re: [Python] Check links
Post by: neusbeer on January 22, 2012, 12:24:25 PM

looks alot like perl.. can be ancestors or such..

Title: Re: [Python] Check links
Post by: Flikka on January 24, 2012, 06:52:05 AM

could you help me with this?


I'm not a python-er-person,
so, how would i run this?

Title: Re: [Python] Check links
Post by: Kulverstukas on January 24, 2012, 08:38:20 AM

save it on a computer. If windows save where you installed Python. Open up the CMD (Terminal for linux/mac). Navigate to where you saved the script, write:
python checkLinks.py

Follow instruction from there.

Title: Re: [Python] Check links
Post by: imation on January 24, 2012, 09:43:17 AM

OH very nice, thak you. this will make life easier