Be careful if setting a large depth and allowing it to go offsite, you could end up drifting on to a lot of other domains. There are also more improvments to the stability of the spider in this release.īy default, CeWL sticks to just the site you have specified and will go to a depth of 2 links, this behaviour can be changed by passing arguments. I've left the words in the case they are in the pages so 'Product' is different to 'product' I figure that if it is being used for password generation then the case may be significant so let the user strip it if they want to. Usage is simple, checkĬeWL now sorts the words found by count and optionally (new -count argument) includes the word count in the output. Found more ways to pull words out of JavaScript content and other areas that aren't normal HTMLĪdded the GPL-3+ licence to allow inclusion in Debian.Īdded a Gemfile to make installing gems easier.Īdds proxy support from the command line and the ability to pass inĬredentials for both basic and digest authentication.Code refactoring by Internationalisation - should now handle non-ASCII sites much better.

CeWL is a ruby app which spiders a given url to a specified depth, optionally following external links, and returns a list of words which can then be used for password crackers such as John the Ripper.ĬeWL also has an associated command line app, FAB (Files Already Bagged) which uses the same meta data extraction techniques to create author/creator lists from already downloaded. Based on a discussion on PaulDotCom episode 129 about creating custom word lists by spidering a targets website and collecting unique words I decided to write CeWL, the Custom Word List generator.

Dictionary file for password cracking