This is a random slice of a random image found somehwere in my blog...
Gibberations by ...
Cyber-Gish | GishTeq
Gibberations RSS feed
598 entries
Wednesday / May 14th / 2003
Scraping the internet...
Alright so I admit it, I have been a naughty boy lately. Desperate to find customers, one often resorts to conniving means, seedy methods bordering on the immoral. Oh well.

You see, I figured why pay one of those companies out there on the Internet tons of bucks for email address lists, or pay them even more to broadcast mailings all over the world when I can do that myself.

I am a pretty smart guy, sometimes. At least that is what I fool myself into believing.

So this is what I did. I wrote a subscription mailing list at GishTeq which allows potential customers to sign up for mailings and newsletters. Later, one may even modify personal settings or opt-out altogether by unsubscribing.

Now the trick is getting people to sign up. Why don't I scan the Internet, collect emails and sign them up myself? When they get the first mailings, they can always unsubscribe.

I created another Perl script which can scan certain sites and scrape off the emails. If possible, it can even log in to subscriber lists and automatically scan these web pages also.

All I do is give the script a site url, start it up and there it goes. Ten minutes later or so I've got my list formatted as CSV so that I can even import it into an Excel sheet for future reference.

You probably won't believe me either if I tell you that with this technique I have successfully harvested more than one thousand warm leads.

Here's a hint (if you know Perl). Use the use LWP::UserAgent module to get the contents of the page and then scan it. See anything interesting, scan deeper via the internal links. Poke around and see if you can find anything interesting. Is there a user id that is passed around? Try all values of that user id from say 1 to 1000 and collect the results.

Sorry, I've been a bad boy. But I want to become famous also.

~ Posted at 04:23 PM | Internet
So far there are 2 comments.

tisk tisk ... the things people will do!
Posted by Stu D. at May 16, 2003 04:39 PM

Yeah, I know. I feel really ashamed about it.
Posted by Kiffin at May 16, 2003 05:10 PM

Add another comment:


Email Address (optional):

URL (optional):
with "http://"

  What's this?

HTML is allowed.Spell checker?

"Life is a great surprise. I don't see why death should not be an even greater one..."  - - Vladimir Nabokov.

From here you can quickly navigate to the previous archived entry («), or to the next archived entry (») or simply go back to the main page.
Have a look below at the comments of others or be the first one to add a comment of your very own for this entry.