The spider/crawler has been upgraded. It now supports gzip and deflate compressed content. This can save bandwidth use on both sides. Check this forum post for more information.