["logo boring","https://github.com/","/plans","/explore","/features","/blog",
"/login?return_to=https://github.com/languages/Haskell/created",
"/languages/Haskell","/explore","explore_main","/repositories","explore_repos","/languages",
"selected","explore_languages","/timeline","explore_timeline","/search","code_search",
"/tips","explore_tips","/languages/Haskell","/languages/Haskell/created",
"selected","/languages/Haskell/updated", "/languages","/languages/ActionScript/created",
"/languages/Ada/created","/languages/Arc/created","/languages/ASP/created",
"/languages/Assembly/created",
"/languages/Boo/created","/languages/C/created","/languages/C%23/created",
"/languages/C++/created","/languages/Clojure/created",
"/languages/CoffeeScript/created",
"/languages/ColdFusion/created","/languages/Common%20Lisp/created","/languages/D/created",
"/languages/Delphi/created","/languages/Duby/created",
"/languages/Eiffel/created",
"/languages/Emacs%20Lisp/created","/languages/Erlang/created","/languages/F%23/created",
"/languages/Factor/created","/languages/FORTRAN/created",
"/languages/Go/created",
"/languages/Groovy/created","/languages/HaXe/created","/languages/Io/created",
"/languages/Java/created","/languages/JavaScript/created",
"/languages/Lua/created",
"/languages/Max/MSP/created","/languages/Nu/created","/languages/Objective-C/created",
"/languages/Objective-J/created","/languages/OCaml/created",
"/languages/ooc/created",
"/languages/Perl/created","/languages/PHP/created","/languages/Pure%20Data/created",
"/languages/Python/created","/languages/R/created",
"/languages/Racket/created",
"/languages/Ruby/created","/languages/Scala/created","/languages/Scheme/created",
"/languages/sclang/created","/languages/Self/created",
"/languages/Shell/created",
"/languages/Smalltalk/created","/languages/SuperCollider/created","/languages/Tcl/created",
"/languages/Vala/created","/languages/Verilog/created",
"/languages/VHDL/created",
"/languages/VimL/created","/languages/Visual%20Basic/created","/languages/XQuery/created",
"/brownnrl","/brownnrl/Real-World-Haskell","/joh",
"/joh/tribot","/bjornbm","/bjornbm/publicstuff","/codemac","/codemac/yi","/poconnell93",
"/poconnell93/chat","/jillianfu","/jillianfu/Angel","/jaspervdj","/jaspervdj/sup-host","/serras",
"/serras/scion-ghc-7-requisites","/serras","/serras/scion","/iand675","/iand675/cgen","/shangaslammi",
"/shangaslammi/haskeroids","/rukav","/rukav/ReplayTrace","/jaspervdj","/jaspervdj/wol","/tomlokhorst",
"/tomlokhorst/wol","/bos","/bos/concurrent-barrier","/jkingry","/jkingry/projectEuler","/olshanskydr",
"/olshanskydr/xml-enumerator","/lorenz","/lorenz/fypmaincode","/jaspervdj",
"/jaspervdj/data-object-json","/jaspervdj","/jaspervdj/data-object-yaml",
"/languages/Haskell/created?page=2","next","/languages/Haskell/created?page=3",
"/languages/Haskell/created?page=4","/languages/Haskell/created?page=5",
"/languages/Haskell/created?page=6","/languages/Haskell/created?page=7",
"/languages/Haskell/created?page=8","/languages/Haskell/created?page=9",
"/languages/Haskell/created?page=208","/languages/Haskell/created?page=209",
"/languages/Haskell/created?page=2","l","next","http://www.rackspace.com","logo",
"http://www.rackspace.com ","http://www.rackspacecloud.com","https://github.com/blog",
"/login/multipass?to=http%3A%2F%2Fsupport.github.com","https://github.com/training",
"http://jobs.github.com","http://shop.github.com",
"https://github.com/contact","http://develop.github.com","http://status.github.com",
"/site/terms","/site/privacy","https://github.com/security",
"nofollow","?locale=de","nofollow","?locale=fr","nofollow","?locale=ja",
"nofollow","?locale=pt-BR","nofollow","?locale=ru","nofollow","?locale=zh",
"#","minibutton btn-forward js-all-locales","nofollow","?locale=en","nofollow","?locale=af",
"nofollow","?locale=ca","nofollow","?locale=cs","nofollow","?locale=de",
"nofollow","?locale=es","nofollow","?locale=fr","nofollow","?locale=hr",
"nofollow","?locale=hu","nofollow","?locale=id","nofollow","?locale=it",
"nofollow","?locale=ja","nofollow","?locale=nl","nofollow","?locale=no",
"nofollow","?locale=pl","nofollow","?locale=pt-BR","nofollow","?locale=ru",
"nofollow","?locale=sr","nofollow","?locale=sv","nofollow","?locale=zh",
"#","js-see-all-keyboard-shortcuts"]Quite a mouthful, but we can easily filter things down. “/languages/Haskell/created?page=3” is an example of a link to the next page listing Haskell repositories; presumably the current page would be “?page=1”, and the highest listed seems to be “/languages/Haskell/created?page=209”. The actual repositories look like “/jaspervdj/data-object-yaml”.
The regularity of the “created” numbering suggests that we can avoid any actual spidering. Instead, we could just figure out what the last page is, the highest page, and then generate all the page names in between because they follow a simple scheme.
So we need to extract the number. Easy enough: the prefix is statically known and never changes, so we can hardwire some crude parsing using drop: drop 32. How to turn the remaining String into an Int? Hopefully one knows about read, but even here Hoogle will save our bacon if we think to look through the list of hits for String -> Int—read turns up as hit #10 or #11. Then, now that we have turned our [String] into [Int], we could sort it and take the last entry, or again go to the standard library and use maximum (like read, it will turn up for [Int] -> Int, if not as highly ranked as one might hope). Tweaking the syntax a little, our final result is:
If we didn’t want to hardwire this for Haskell, we’d probably write the function with an additional parameter and replace the Int with a runtime calculation of what to remove:
So let’s put what we have together. The program can download an initial index page, parse it, find the name of the last index page, and generate the URLs of all index pages, and print those out (to prove that it all works):