It all started with a simple post from Jason Haley while I was on vacation with my family at Emerald Isle.  The post was about creating a link blog aggregator to show the days top picks and diamonds in the rough with or without having to sift through everyone's mention of X.  Like I said, I was on vacation and trying really hard to stay disconnected.  I managed to slip in a few minutes here and there while my youngest daughter took her naps throughout the day and played a bit with Hpricot trying to parse some feeds.

I put things on hold, finished vacation and got caught up with a couple other things.  Finally I came back and decided to crank out a 1.0 version (no beta's here).  The result is hushchamber.com

It's another ruby on rails project for me with some interesting challenges.  I'm using backgroundrb to do all my asynchronous and background processing (mainly fetching new feeds from RSS and parsing those feeds for new links).  I'm using god to monitor backgroundrb and restart it if one of the workers goes down, this normally happens for me with a timeout with open-uri (I think, I can't catch the error in a rescue for some reason).

So what were the challenges?

Parsing was a big one.  Not that it's difficult with Hpricot but some blog clients produce better markup than others.  I'm trying to get only link mentions and not links to other stuff (like the link bloggers blog, or job search sites, or events that are repeated for up to two weeks).  In some instances I had to strip out some content before I could successfully parse it.

Links, surprisingly were another one.  Depending on how the linkblogger linked to content--sometimes through a feedburner feed, sometimes directly to the resource on the web, some with URL shortening services--produced different URLs.  The challenge here was getting all 3 links to the same resource to behave like the same resource.  I ended up having to "resolve" the URL to it's original web form and use that as the unique identifier for the link.

Titles were another issue, titles for links that is.  I started using the inner html for the anchor tag as the text/title for the link itself.  So in the following Example I'd grab "Jeff's Blog" and set that as the text to use for the link.

  1: <a href="http://thequeue.net/blog">Jeff's Blog</a>

The problem is, some titles have <font> and <strong> tags in them, and I really didn't want that.  Other times it's just that link bloggers description of the resource which may or may not make sense out of the context of the blog entry itself.

I played with the idea of going out to the resource directly and using hpricot again to grab the <title> attribute on the page.  I actually did that but 90% of the sites I tried that on have the real title plus some other information about the site as well.  It's really bad when it's a Code Project link, the title++ portion is something like an extra dozen words.  I ended up going back to just using the linkbloggers inner html, and it's first in wins.

Top links and diamonds in the rough

Surprisingly there weren't as many links mentioned from a bunch of linkbloggers as I thought there would be.  Even on jQuery Monday there was a lot of talk about jQuery and Visual Studio and ASP.NET MVC but they were all to different posts.

Which brings up another interesting suggestion, or thought, to display links grouped by tag.  I've played around with going out to del.icio.us (this is still faster than their new URL for me to type) and grabbing the top tag for a given link and grouping by that tag.  We're left at the mercy of the initial taggers of a resource but most of the time they're close.  Though it doesn't help for a completely unrelated bunch of links.

So, check it out: http://hushchamber.com

There's also an RSS Feed for it that you can subscribe to and get the last 30 days or so worth of links, if old posts come in you do get updates to RSS.  let me know what you think by using the feedback button:

In a previous post I described how I'm keeping up with APRESS' eBook Deal of the Day.  At the time I was using a Firefox extension, Update Scanner to watch for changes to the web page.  It worked well enough, but it still required me to physically go out to the page and I've noticed a couple quirks.  Since then I got excited about a tangent, zenhabits Twitterbot Challenge.  I never actually participated in the challenge as the Twitterbot thread took me in a different direction.

While exploring some of the different uses people have been using twitter for I saw a w00t twitterbot that tweets w00t items, handy in a w00t off.  That wasn't far off from what I wanted from APRESS (I would have been happy with RSS, but I'm using twitter regularly enough that it's a decent solution).  All I need to do is go out to the daily deal page, scrape it for the book name and link, then tweet it.  How hard can it be?

Not that hard, really, about fifteen lines of ruby code and a few gems:

  1: #!/usr/bin/ruby
  2: 
  3: require 'rubygems'
  4: require 'hpricot'
  5: require 'shorturl'
  6: require 'twitter'
  7: require 'open-uri'
  8: 
  9: doc = Hpricot(open('http://www.apress.com/info/dailydeal')) 
 10: deal = doc.search("//div.bookdetails")
 11: book = (deal/"h3/a").first
 12: description = (deal/"p")
 13: details = (description/"div")
 14: (description/"div").remove
 15: 
 16: root_url = 'http://www.apress.com'
 17: book_url = ShortURL.shorten(root_url + book.attributes['href'])
 18: book_title = book.inner_html
 19: 
 20: tweet_start = "[#{Date.today}]: "
 21: tweet_end = " (#{book_url})"
 22: book_title_shortened = book_title[0, 140 - tweet_start.length - tweet_end.length]
 23: tweet = tweet_start + book_title_shortened + tweet_end
 24: 
 25: twitter ||= Twitter::Base.new("ApressDailyDeal", "my_super_secret_password")
 26: twitter.post tweet

So, let's break it down.

Line 1 is there because I'm going to set this to run as a cron job later and I'd rather do ./dailydeal.rb than ruby dailydeal.rb.

Lines 3-7 bring in some gems (similar to perl's cpan packages).  open-uri to get the page, Hpricot to scrape it, short-url to make the link to the book shorter and twitter to tweet.

Lines 9-14 could probably be tightened up, but I rather like the "clarity over cleverness" meme.  Surprisingly, APRESS' page was rather easy to scrape which means it was well marked up, using classes appropriately for semantic meaning (div class="bookdetails" actually held... book details). 

I stitch the URL together in 16-18 and call out to short-url's shorten method.  It supports a bunch of services, rubyurl is the default and with tinyurl not working for me, I went with it.  I actually forked the project on github to add support for is.gd (every character counts with twitter) but I'm waiting for a gem update before I switch to that.

In lines 20-23 I assemble the tweet.  I broke it into a few parts because I need to make sure the message length doesn't exceed 140 characters.  The date and URL to the book are more important than the entire title of the book, so I truncate that if necessary.

The last lines are just sending the tweet.

On my linux box I used crontab -e to enter the following job:

  1: 30 08 * * * /home/jeff/development/apress_twitterbot/dailydeal.rb

So at 8:30 every morning, give or take a bit of drift either way, I'll tweet off today's new Daily Deal:

The original page:

 So, if you're interested in getting APRESS' Daily Deal tweeted to you every morning (EST) follow my bot: http://twitter.com/ApressDailyDeal

eBooks are becoming a lot more interesting with the introduction of the Plastic Logic reading device (http://www.plasticlogic.com/)

APRESS has been discounting a single eBook to $10 for a 24 period for a couple months now, that I know of.  They're calling it the eBook Deal of the Day:

At 12:01 AM PST, a different Apress or friends of ED eBook will be priced at $10 US for a 24-hour period. Check back everyday for a new deal. If you have any further questions, please contact support@apress.com.

Awesome, $10 for an ebook is a good deal, even if I can't read one cover to cover digitally I can still search through it as a reference.  I end up just printing ebooks, which probably defeats the purpose, but oh well.  Maybe when the Kindle or another reader is up to the job for technical books it'll be a different story.

Anyway, I like to keep my eye out for a deal so I'll just subscribe to the RSS feed and check my reader for ebook goodness.  Except there is no RSS feed.  Come on, APRESS, this just screams for a feed.  I dropped an email to everyone I could think of, my UserGroup liaison, her boss because she was on vacation and had a responder, support and suggestion links on the website, everyone.  Still no RSS feed.

So what's a geek to do?  Try one of the several "create a feed for non-RSS enabled sites" services and get the information that way.  I tried a couple but either I or they were obtuse and impatient and I couldn't get what I wanted.  In retrospect, I could have used that time with a bit of Hpricot and probably had exactly what I need.  I might do that later, look for another post on that.

I then turned my attention to the always reliable Firefox extension community and found Update Scanner.

After you install the extension, press ALT+U or select "Update Scanner" from the tools menu:

That will open up a sidebar with pages you're interested in scanning for updates.  Click on the far left button to start scanning a new page:

It's smart enough to fill in some defaults for the active tab in Firefox.  If that's not what you want just make the changes.  Title could be cleaned up a bit: "Apress eBook deal of the day" maybe?

I put the threshold fairly low for changes as they only change about a paragraph of text every day. I'm also only scanning their site every day, let's try to be courteous users and not hammer their site.

Now all you have to do is wait until tomorrow for something that looks like this and you'll know that there's a new deal (or that something has been updated if you're using this for other pages).  Update Scanner can also open all updated pages in new tabs, handy if you're diffing the net during morning coffee or something.

So why the post?  Most of the deals I've seen so far have been for older material, this doesn't mean they're not good, some texts are classic and their content might always be relevant, but books about version 1 of the .NET framework probably aren't.  Today's was actually published mere months ago on SQL Server 2008 XML (hopefully not the single row, single column XML blob that I've actually heard people talk about as a good idea! ).  In the future if I'll pass along other "good deals" in my opinion.

I have no affiliation with APRESS, just think of this as a PSA.

The ASP.NET MVC Team released Preview 4 today on codeplex.  Scott Guthrie is planning a two part series to introduce it, part 1 is here.  Phil Haack's follow up is here.

I just found out that I have been rewarded an MVP for ASP/ASP.NET.  It's an honor to be part of such an awesome group and the .NET Community.

I can almost here the angels singing in the background.  ReSharper 4.0 has finally been released and finally supports all the new C# 3.0 features.

I spend a lot of my time helping a customer maintain a ASP.NET 1.1 application written in VB.NET, that somewhat muffles (or muzzles) those angels...

I had a couple people ask me what the zoom tool I used during my MVC talk was, it's ZoomIt from SysInternals.

You can find it here.

Just a quick reminder for everyone that registered, NoVa CodeCamp 2008.01 is this Saturday (May 17, 2008) at the Microsoft Office in Reston, VA.

Vitals:

Directions
Schedule

The Nova CodeCamp 2008.01 schedule has been released, you can find it here.  There may be one or two changes, but only one of them should be major.

The low-down:

When: May 17, 2008 from 9a - 6p (registration begins at 8:30)
Where: Microsoft Offices in Reston, VA (directions)
How: Gracious support from our contributors, speakers and volunteers.
Why: Because we don't get enough of this during the week... Because all of our speakers and volunteers are passionate about community and want to give back.
How much: Free (and we'll feed you pizza!)

Where can I sign up?  Register here.  We have a limit of 200 and we're almost full, so register now.