View On Black flickr bookmarklet

I’ll admit, I’m a fan of flickr and upload all of my pictures over there. One thing that I don’t like about flickr though is the default white background which unfortunately can make the photos look a little dull. So, generally I would include a link in the description to http://www.bighugelabs.com/onblack.php so that the viewer have an option to view the photos on black. Unfortunately, this meant that every time I upload a photo, I had to manually copy over the link to bighugelabs into the description. Since I am a sucker for redoing same stuff over and over again, I decided to automate the entire process  and ended up writing a simple javascript bookmarklet. Now all I need to do is to navigate to my photo details page and just click on the bookmarklet and voila, the link to “onblack” version is automatically added in the description. Below is the bookmarklet just in case you are also a fan of “onblack”:

javascript:var%20evt%20=%20document.createEvent(&quot;MouseEvents&quot;);%20evt.initMouseEvent(&quot;click&quot;,%20true,%20true,%20window,%200,%200,%200,%200,%200,%20false,%20false,%20false,%20false,%200,%20null);%20var%20cb%20=%20document.getElementById(&quot;meta&quot;).getElementsByTagName(&quot;div&quot;)[0];cb.dispatchEvent(evt);var%20reg=/\/CHANGE-THIS\/(\d+)\//;var%20y=window.location.href.match(reg)[1];document.getElementById(&quot;meta&quot;).getElementsByTagName(&quot;div&quot;)[0].getElementsByTagName(&quot;textarea&quot;)[0].value='<a>View%20On%20Black</a>';document.getElementById(&quot;meta&quot;).getElementsByTagName(&quot;button&quot;)[0].click();void(null);

The CHANGE-THIS should be changed to your flickr's NID or your flickr friendly Url i.e. whatever you see in the address bar of your flickr photostream for e.g. if your flickr photo url is http://www.flickr.com/photos/xyz you should replace CHANGE-THIS to xyz. Do keep in mind that this bookmarklet will replace whatever is there in the description field with just a link to “OnBlack” version with the text “VIew On Black”. Also, I have only tried this on Firefox and it works in almost all the scenarios that I tried.

Contextual spelling suggestions using solr SpellCheckComponent

One of the problems that I faced while trying to implement the spelling suggestions (or "Did you mean") in solr was with phrase queries. If a user types in multiple terms and say one or more of the terms are mistyped, solr provides suggestions on the individual terms (using edit distances) in isolation; and then collates the results of the top most suggestions and returns back as spellcheck.collation. It's all dandy if the user just types in 1 search term or if your searches are OR searches but in case you do an AND search (or some other form of AND using the dismax), there might be instances when the spellchecker's collation actually returns 0 results; and thus resulting in a bad user experience. Let me try to explain this with an example:
Assume that the user types in "chicen fayita sadwich" and you have just two documents in your index:

Doc 1 ==> chicken fajita
Doc 2==> veg sandwich
Now, since solr/lucene treat the terms in isolation what you get in spellcheck.collation is:
"chicken fajita sandwich". Unfortunately, if you have a AND search and the user does click on this "Did you mean" link, it would result in a query (chicken AND fajita AND sandwich) resulting in zero results.
So, how do you solve this? One way to solve this is to use Shingles for creating your "spelling corpus", the only trouble with Shingles is that the number of suggestions generated is bound by the "MaxShinglesSize" and thus if you set your MaxShingleSize to say 4, you only get suggestions up to 4 terms.
Another cleaner albeit slightly slower approach is to extend the solr.SpellCheckComponent, and fire another solr query on the collation itself; if the suggestions' results are greater than 0 (or some other threshold), you return the collation back else try with the second set of suggestion (or just blank out the collation in case you don't want to fire multiple queries). This is what I did (though I am not 100% sure if this is the right way of firing solr queries): extend Solr.SpellCheckComponent, & hook onto the processRequest method to get the handle of ResponseBuilder object (so that you can then get SolrRequest and SolrSearcher objects from this). Override toNamedList method and in that get the collation string, fire another solr query using SolrSearcher and check the results' count; if the suggestion.Count > originalQuery.Count * THRESHOLD, let the collation be as is else blank it out. The guts of this is the overridden toNamedList method, which is below in case somebody is interested:

    protected NamedList toNamedList(SpellingResult spellingResult, String 
origQuery, 
            boolean extendedResults, boolean collate) 
    {
        NamedList result = super.toNamedList(spellingResult, origQuery, 
extendedResults, collate);
        if(collate){
            String collation = (String) result.get("collation");
            if(collation!=null && collation.length() > 0 && builder!=null){
                //fire a query and get the results
                try {
                    //only add spelling suggestion in case results are less than 
some threshold
                    int hits = builder.getResults().docList.matches();
                    if(hits>MIN_THRESHOLD){
                        result.remove("collation");
                        //result.add("collation", "");
                        return result;
                    }
                    SolrIndexSearcher searcher = builder.req.getSearcher();
                    QParser qp = QParser.getParser(collation, "dismax", 
builder.req);
                    NamedList params = new NamedList();
                    params.add("rows", 0);
                    params.add("omitHeader","true");
                    SolrParams localParams = SolrParams.toSolrParams(params);
                    qp.setLocalParams(localParams);
                    Query q = qp.getQuery();
                    TopDocs docs = searcher.search(q, 1);
                    int suggestionHits = docs.totalHits;
                    //try to get hits for this query
                    log.info("current hits:" + hits);
                    log.info("total number of hits:" + suggestionHits);
                    if(suggestionHits <= hits*MULTIPLIER){
                        //remove the collation
                        result.remove("collation");
                        //result.add("collation", "");
                    }
                } catch (IOException e) {
                    log.error(e.toString());
                }
                catch (ParseException e) {
                    log.error(e.toString());
                }                
            }
        }
        return result;
    }

Implementing Autosuggest in ASP.Net using WCF Rest service, JQuery and solr

I finally had to do give in and implement autosuggest (aka autocomplete, aka predective text, aka look ahead) on our free text search box ala Google. The requirement was simple: implement autosuggest based on previous user typed in free text queries which return some results back (as it would not be a great UX to suggest a term which returns 0 results back). Implementing this required: a. persist user typed in queries somewhere b. write an autosuggest component based on a. and glue it to the UI to return suggestion list to the user.
Part I: Setting up solr index
Since, we already use solr 1.4 and I am not big fan of making database do anything more than just persist/retrieve data, I knew I would rather write my autosuggest component utilizing solr.
As far as I know, there are 3 ways (apart from wild card search) of achieving autosuggest using solr 1.4:
1. Use EdgeNGrams
2. Use shingles and prefix query.
3. Use the new Terms component (in solr 1.4, the terms component does not have the regex support, which means it can only do a "begins with" match).
After a bit of contemplation and research, I decided to go with using NGrams. Basically, the decision was made by the process of elimination:
a. shingles & prefix query: I know AOL does this way but unfortunately there's very little documentation online around shingles and how you go about using it; I figured; I would be groping in the dark if I went ahead with this.
b. Terms component: Even after applying the regex patch from JIRA, I felt there is lot more sanitizing of input that needs to be done (things like if the user types in multiple terms like: "joan of", I need to ensure that I replace the whitespace with \s etc). I somehow felt it's easy to break this in case a user types in slightly odd search terms.
Having decided on going the EdgeNGrams route, all I needed was two fields in schema file. Here's how the solr field definition for the "suggest" field looks like:

<fieldtype name="suggest" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory"
minGramSize="2" maxGramSize="50" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>


The suggest field holds the NGrams, whereas the query field stores the query verbatim (apart from lowercasing it). This allows us to return the query back to the user in the suggestion list based on what he typed.
Part II: Building the index
Part I of the problem solved, now we have the solr index which would generate the NGrams based on the query that it needs to index and also store the query that would be returned as part of the suggestion list.
Now, we need to figure out how do we build this index i.e. how we get the user typed in queries into this index. Instead of indexing each and every query the moment the user types it in, I implemented a simple lazy writer (in fact two levels of lazy writer). Every time the user types in a query which return results, I store that in a cached dictionary. I hook on to CacheItemRemovedCallback and persist the aggregated queries into a database table, the db table also holds a DateModified timestamp column which is updated to getdate() for newly added items. We then have a offline task which runs on a nightly basis and indexes all the newest queries to the solr index. Part II solved, now we have the solr index built and ready to serve "suggestions".
Part III: gluing it with the presentation layer
There are two bits to the presentation layer:
a. Hooking the solr index to get back the suggestion list as JSON.
b. The presentation itself aka UX.

Since, we already use JQuery on our project I decided to go with Autocomplete plugin for UX. The plugin again lacks in-depth documentation but it wasn't very hard to get it working. Instead of calling the solr index directly from the autocomplete plugin (which would have the serious after-effect of exposing our solr endpoint to the public and also adding solr dependency to the UI layer), I decided to write a WCF REST shim around the solr index. Adding a shim/decorator on top of the solr also gave me a place to intercept the calls to the solr and sanitize the user input (basically escaping the Lucene special characters) and do some heavy duty caching. That's where I had a problem: WCF services in 3.5 don't have caching support built in, you need to install the REST Starter Kit to get the caching support but once I had it installed I was able to get caching working with my WCF REST service.

After thoughts
We haven't rolled out this implementation yet, so I don't know in terms of performance how it's gonna go but at least during my limited testing, I have been able to get quite fast response times. By the way, SOLR-1316 seems quite promising too and I've been watching that thread with quite some interest: it talks about implementing the terms component as a ternary search tree so that should be quite fast.

solr and indexing Chinese text

So, I've been kinda bugged with trying to get our search functionality upto the mark for the Chinese (I also have the other JK in the CKJ to worry about, but more on that later) language. As some of you would know, the biggest issue with the CJK language is that there are no word boundaries unlike the latin languages. Due to the missing word boundaries, the issue is how do you tokenize the text? There are some brute force ways of tokenizing (tokenize on uni-gram) and some middle of the road approaches (tokenize on bi-gram, which is what the lucene's built in CJKAnalyzer does). We started with the CJKAnalyzer but unfortunately figured out that it just doesn't cut it esp. due to the fact that CJKAnalyzer does not let you search on single character tokens (as it only tokenizes on Bi-grams). So, I decided to bite the bullet and upgrade solr to 1.4 (I guess I should post my experiences on that too some time) which also means that now we have lucene 2.9 to play with. One of the contrib package with Lucene 2.9 is the smartCn analyzer (http://issues.apache.org/jira/browse/LUCENE-1882), and I decided to give it a shot. The smartCn uses a built in dictionary and Hidden Markov model (I have no clue what it means) to "smartly" tokenize the chinese string. SmartCn analyzer did solve the problem of single character searches and might get us through the next release but I still feel it still doesn't solve the problem to my comfort level (the multiple character searches return less result using smartCn than the CJKAnalyzer). I guess, one way to solve it is to seed the built in dictionary with chinese terms which are relevant to our site, unfortunately the built in dictionary seems to be serialized into some binary format from a java class which I am having some hard time to decipher. I need to dig a little more deeper into this and maybe shoot out an email to the contributor of this package to see if there is some easy way of seeding the built in dictionary.

oops, something when wrong...

So, there's a lotsa buzz about Google Reader Play and I was tempted enough to try it. But when I when (sic) there, all I got was a "oops, something when wrong..retrying" message:

Well, maybe that's what beta means these days...

fly, fly away

I still remember when I was a kid growing up in mid 80s, I would stand on my terrace and watch Woodpecker digging holes into the trunk, Owls sitting on the tree top. Watching hoards of vultures scavenge carcass on the road side was a sight everyone in those days would be quite familiar with. I would be in awe of all these wonderful creatures. Many years have passed since then and along with them most of the birds have disappeared or sighting them has become a rare phenomenon. Even, a "common" sparrow is not so common these days, all I get to see these days are crows, pigeons and kites, if you want to watch some exotic birds like the hill mynah, vultures, owls you need to go to places which are "reserved" for these creatures by us (read bird sanctuaries) or go to places where humans have not yet cut the trees and built concrete jungles. With the rate at which the birds are disappearing or are being confined to "reserved" places, I guess, the children of tomorrow would only have pictures of these birds without having a chance to see for themselves.

just my imagination

It's been a while as usual since I posted last and I guess it has something to do with how I feel these days. I think writing posts require pretty decent amount of imagination and creativity (unless you want to write just for the sake of writing something), and off-late I don't seem to have either; it's not like that I don't have anything to write about; it's just that I have don't have enough creativity to elaborate it into a meaningful post. I've been thinking about writing on some of the technical stuff that I learned in the recent past (mostly the hard way), some SEO stuff, some photography tidbits, some gear lust but every time I feel like writing something; I just draw a blank. The other thing that I always wanted to try my hands on was short stories, I have a quite a few themes in my mind (some callous, some not so) but again just don't seem to have enough steam to get those into something concrete. I know I need to be a bit more frequent with my posts (just for my own sanity) but somehow I can't get my darn act together!

Photography tidbits

One of the first thing that bites you if you're into photography is what subject to photograph? Do you just take pictures of the same ole' flowers, monuments or do you try to be different. The problem with trying to being different is finding out all those things which haven't been captured before. So, you hunt for those places/things (either mentally or physically) and then give up and in the end take the pictures of the same subjects which have been taken umpteen times before (butterfly on a flower, a flower petal, sunsets etc). You still believe that at least your photographs offer something different, maybe, a new perspective, a new *vision* (basically, all the things that you try to convince yourself with). Then, you get into some equipment acquisition syndrome and dream of owning the best in the class (read the most expensive); Zeiss & Leica lenses, Nikon D3X, Hasselblad: if something is so freaking expensive it gotta be good, right?

I am one of the 3 people in my country (the other two haven't been vociferous about it yet) who owns a 4/3rds system (a Olympus e-510), and honestly apart from a tiny-winy viewfinder; I haven't had too much to complain about the camera (I generally take all the blame for out of focus, horrifically shaky pics). Quite a few people have asked me why I invested into a 4/3rds system; well; just like my any other investment it wasn't something that I carefully thought about but was just an impulsive buy (I didn't even try it or any other brand before buying). It's a funny thing that I always believed I don't need a dSLR and was happy with my P&S (a Canon A95) till I decided to go for a trip to Corbett National Park where I had an opportunity to capture a tiger in the wild for the first time and boy, did I manage to capture it with a 3x optical zoom of Canon! I think sometimes you just need a better equipment to bail yourself out. Anyway, has the dSLR made me a better photographer than before? No, definitely not but it does allow me to experiment a bit more and push my own creative limits…it reminds me of a saying that I read long ago:

The difference between a picture and painting is that

a picture is a frozen captured moment of true life,

whereas a painting can be anything imagined in your mind.

..... someday, all these paintings will become pictures.

where art thou?

Been a while since I wrote something, have been busy off-late with doing nothing. I guess, nothingness is a hard feeling/habit to get out of! Anyway, some time back I started playing with a cooked WM6.5 on my phone and I think though there still are miles to go but it's a step in the right direction by MS (finally a WM version which doesn't require you to pull out your stylus!). Anyway, one thing that I found missing in titanium was the task plugin so the developer in me hacked together something (more on how I did it later, given that it was my first attempt at building something for the mobile device) which did what I wanted and it now is available on xda-developers with a very original name of taskanium.

Apart from that I have revitalized my love for photography (albeit in a very small way) and have been bitten by LAS (Lens Acquisition Syndrome). I also think, I should start something like "How-I-Took-It" posts, where I can just dissect my photo and the technique that I used to capture the shot (Ok, I am no pro but I guess I have the basic concepts of photography clear).

solr and locale sensitive sort

Another of those meaningless exercises:
============================
We've been using solr on our asp.net project for quite a bit now and have been quite pleased with it, everything went great till we had to support character based language (Chinese to be more specific). Based on my research, I decided to use CJKAnalyzer for analyzing and tokenizing and the results returned were generally acceptable, the only issue being the results were not sorted in any rather random order if I asked solr to sort results alphabetically. After couple of hours of hunting solr user groups and looking at the solr source, I realized that solr does'nt provide any support for locale sensitive sorting. Since finding out the problem is half the problem already solved, all I had to do was extend from solr's built-in StrField; add support for another attribute (locale) in schema.xml and pass that to new instance of SortField and voila, solr was happily returning the results sorted by pinyin. In case you are interested, below is the discussion around the problem and the JIRA issue for solr (which has the new locale sensitive custom java class):

http://www.nabble.com/CJKAnalyzer-and-Chinese-Text-sort-td22374195.html
https://issues.apache.org/jira/browse/SOLR-1073

do we need guitars?

I was just reading the "starting over" post on Mars Need Guitars, a blog I used to follow earlier to check out music before it disappeared into oblivion. It's funny what depression can do to you, it can slowly eat away all that is inside of you and even before you know what hit you, you are already spiritually dead. A lot of things in that post reminded me of myself, someone who too is trying to get past every day; and (luckily) somehow succeeding at it. I just wish I could start afresh and do things that I want to do: write some short stories, write the book that always I wished to write, get away from this meaningless stuff and just break free. Maybe, one of these days, I will.

Questions

A new year and same day to look back at life and think about where I am. Over the past few years a couple of questions have haunted me:

1) Where am I?
I don't think I know where I am in terms of my mental & emotional state; the scary thing is that I don't even care any longer.

2) Am I happy doing what I am doing?
No, I am not happy doing what I am doing either professionally or personally. Life has just been meandering along all these years. Earlier, I would get all worked up and try to change things that I wouldn't like but off-late it's more of sorta resigning to the facts...I don't even feel like doing anything to change my life and/or its course.

3) Do I know what I want?
Everyone will tell you that they are not getting what they want, the trouble is very few of us are even aware of what we really want. I have no guilt in admitting that I don't know what I want in my life. I can tell for sure what I DON'T want, but I don't know what will make me happy, any longer. It's just that sometimes, I feel like running away; don't know what I want to run away from, perhaps myself.

4) Have I changed over these years?
Yes, I have and quite a bit. I don't feel that strongly about anything. I used to fight for what I felt was right; I don't now. I have no problems in doing what others want me to do cause I don't feel strongly for or against anything. I believe feelings are strongly connected with your emotions and since I spent so many emotions over these years; it looks like my emotions have run out. I just go through the motions now; don't want to change anything or more rightly put; don't have any strength and motivation to change anything. Why shall I change something which doesn't matter to me, anyway?

5) Do I deserve more?
I don't know, anyway it doesn't matter what I think. My life has always been dictated by what others wanted in their life; perhaps I've let people come too close to me and yes, if I get a second chance I would still do the same thing over & over again without event a second thought.

bad programmer, no cookie!

One of things that I'd been thinking of while trying to optimize the website we've been working on is cookies. One of the issue with a non-www site is that it generally is the root domain, so any cookie that is issued by the root domain will result in the browser sending that cookie with all the subsequent requests (even requests to a sub-domain and for static content). So, let's say you have a site hosted at http://example.com and all the static content is served through a sub-domain (say http://cdn.example.com) by a CDN, any cookies set by example.com have to be sent to cdn.example.com, even though the static content hosted at CDN doesn't need the cookies set by root domain. Most of the perf guides that I've read, generally recommend using a cookie-less domain , by hosting the main site on a www subdomain or by using a new domain altogether for serving the static content. Using a www or non-www version is sometimes a corporate choice (which might be influenced by the company's branding; as in our case), and using a different "cookie-less" domain might run you into XSS limitations of the browser. So, wouldn't it be nice if I could provide some kinda hint to the browser as to when not to send the cookies using some meta tag. For instance, if I want to exclude a sub-domain from receiving any parent domain's cookies, I could do:
<meta name="no-cookies" content="cdn.example.com,cdn1.example.com...">
And, in case I want to exclude certain file types on the same parent domain itself from receiving any cookies, I could do:
<meta name="no-cookies-for-files" content="jpg,js,css...">
The second meta tag will ensure that browser doesn't send back the cookies for any (static) content which doesn't need the cookies even though the content is served by the same domain. The no-cookies-for-files has to be file extension based instead of MIME based cause to figure out the MIME type, the browser has to make a request first. We can extend this idea even further; but I guess adding these two meta tag support (call it anything that you wish) would be great!
thoughts?

bloomr: what's blooming?

Very few people know that I love gardening and I love flowers (one of my new found interests while I was on a 8 month job break). One of the challenges for a budding gardener is to find out what all is blooming during this time of the year, and though you can make a dash to a nursery; lazy bums like me prefer the e-way! So, my way of finding the (good & not so common) flowers is by scurrying flickr to find recent blooms in my city. This method has served me well but since I've been completely bored off-late; I decided to automate the entire process.
Enter popfly:I had played with popfly when it was launched; and couple of days back I decided to log in to see if there was any new features/improvements to it; first thing that I figured out was that none of my blocks were working! Fixing them wasn't tough and since I had long ago written a flickr block to retrieve photos from flickr; I decided to re-use it for my flower search. The logic is quite straight-forward; the block expects the city name (actually, a place) as a user input; hits flickr to get the place id for the user entered text and then hits flickr again with a hard-wired query (currently set to: flower+bloom). Since, I would most probably be only interested in finding the recent blooms (instead of flowers blooming last summer), the code also limits the photos to the ones which were taken in last two months (current month-2). Finally, the block wires the output to a display block to display the flowers in some kinda slide show. I ran into couple of issues with the wiring of display block I decided to use (ImageSlideShow): it wouldn't clear the images before starting a new slideshow; this would mean if I search for say Boston first, and in the mid of slideshow; decide to search for Seattle; the seattle flowers are appended to the boston list! Fixing this issue meant, ripping off the ImageSlideShow block; exposing the clearImages method and calling it in the glue code between flickr block and ImageSlideShow block. Once, the mashup was all set; I decided to create a web-page around it in popfly; unfortunately the html generated is not the best (understatement) and I haven't felt like tweaking it (yet). I generally don't like iframes but that's what you get with popfly and I still haven't figured out a way to change the background of my mashup. Anyway, in case you are interested in giving [i]bloomr^tm a twirl; you can give it a shot here: on popfly or here

A thousand times over

Six years ago; on this day i'd asked myself a question, a question which I answered without any hesitation...an answer which changed my life and is more or less responsible for where i am today. So would i now have come up with a different answer, given how life has been all these years? No, i still would have given the same answer, a thousand times over.

Building a profanity filter with Asp.net

One of the disadvantages of a web site which relies heavily on user generated content is unwanted content and profanity. Since, there will always be more users then moderators you will have to rely on community policing to bring down unwanted content. But, rather than only rely on community policing (which might work well for a web 2.0 site), you might also want to build a basic profanity filter for your web site or the blog (where there can't be any other moderator except the blog administrators). On the current .net 2.0 site that we worked on we had to build a basic profanity filter using a custom dictionary; since, the only place where you would want the profanity filter to work will be input text elements and textareas (asp.net textbox), I could think of three approaches:

1. Sanitize the user input every time and on every page i.e. remove the unwanted text on every page and for every textbox on the page by calling a method in class library . This approach is the least scalable of all as it requires every developer to call the required method diligently and unnecessarily adds to the code bloat.
   
2. Extend the textbox control & create your own control which internally calls the sanitize method in the getter of overridden Text property; the developers are only required to use the extended textbox instead of the base textbox on their pages. You can also in this case add additional properties like SanitizeText (bool) which can be set to false; in cases where you don't the text to be sanitized. Also, you might want to check if the textbox is not of Password type before running it through your profanity filter!
   
3. Use tag mapping to substitute the base textbox with the extended textbox; tag mapping works great when you are already in the middle of your development cycle and have to implement such logic after-the-fact.
   

   
   

Given that we had to implement profanity filter after the fact; we went with approach 3 and so far it has worked great for us. By the way, the profanity filter was built in-house with a custom dictionary (since we needed it for multiple languages) by running the user entered text by the custom dictionary and doing a simple Regex.Replace.

learn, learn, learn!

the beauty of spirituality and philosophy is that anything and everything can be attributed to life's lessons that one has to learn in order to grow and evolve. The more painful the physical manifestation of the event, the higher the lesson that you've learned. This model works great for one or two events where you can try to see the brighter side of things but soon you get tired of convincing yourself that everything is for good! the fact is there are some events in life, which no matter how hard we try turn to them around; they always come around to haunt us and remind us that life at times can be hurtful and there is no point in fighting your feelings or tricking your mind into not feeling the pain: it's alright to fall apart, sometimes.

building high performance asp.net site

Recently, I was involved with building a web 2.0 site on ASP.net where the most important criteria was performance! The pages had to load supper fast (Average Response Time <2sec); style="font-weight: bold;">Caching: The biggest bottleneck in any website is database access; so you need to cache data aggressively: also, unless you are building a site which requires real time data you can live with showing stale data to the user (we make sure that the updates always pull the real time data so that you not displaying the stale data on the update page to the user). Caching alone (esp. lookup data) can increase the performance of your site multi-folds and at times; this alone can be enough to get to decent response times. The only caveat with caching in ASP.net is in a distributed environment; unfortunately in ASP.Net 2.0 there is built in way of having distributed cache objects i.e. every node in the cluster has it's own copy of cached data which can be out of sync' from other nodes. One way to avoid this by either using a third-party cache provider like memcached (I haven't personally used it so not sure how it gels with ASP.net) or upgrading to .net 3.0 and using Velocity; or by making your load-balancing sticky i.e. a user always hits the same node on the farm for his session.
Web.config debug attribute: Make sure you set the debug="false" on production web.config; you don't want debug symbols to be loaded on production environment.
Ajax: Use Ajax where appropriate to reduce the perceived response time; but do keep in nmind that one of the biggest issue with using Ajax for ASP.net (aka Atlas) is that you can't use a CDN to serve the ScriptResource.axd which means all the javascripts are served by your servers (and there are quite a few of them around 9!); this means that if your servers are located in the US and the user is accessing your site from China; he will have to make 9 requests just to download the javascript files and this greatly increases byte download time.
Offline Tools: There are lot of things that can be calculated/processed asynchronously; you can easily offload these processes to an offline tool and reduce the load on web servers (for instance calculating the tag cloud can easily be done once in a day or week as part of a offline tool).
Lazy DB Writes: Keep in mind that db writes are the slowest so it might be a good idea to do some db operations in a batch; the batch data can be written to a flat file or a MQ or even to a distributed cache environment where it can be picked up by the offline tool.
Page Compression: Since there is no direct way of g-zipping the pages in IIS; people don't even think of compressing there pages; compression can drastically reduce your page sizes. Here is the link which provides directions to setup page compression in IIS 6.
Optimize Images and JS: Images should be optimized for the web and js files should be minified.

That's all that we have done so far to achieve a pretty decent response times across our site; what do you believe that I missed out on?

IDisposable

C# trivia: what happens if you wrap an object which doesn't implement IDisposable within a using block?
Ans: Nothing, the code compiles alright and runs just fine; the problem occurs if you assumed that only IDisposable objects can be wrapped within a using block and once the object goes out of using block, the Dispose() on it would be automatically called and hence freeing up your resources; just like I assumed when I wrapped a XmlDataReader within the using block; (wrongfully) thinking that the connection associated with it will automagically be closed once xmldr goes out of scope. Unfortunately; everything worked just fine till one day I started getting Connection pool exceptions and realized that any connection used by xmldr is not getting closed and had to explicitly close them within a finally block! Ideally, only objects implementing IDisposable should be allowed to be wrapped within a using block and you should at least get a compile time warning (if not error) in case you try to place a non-Idisposable object.

Aside: I guess you can wrap me within the using{}, cause I-sure-am-Disposable!

my thoughts on solr

Recently, we had a requirement to provide some advanced searching capabilities for a public asp.net web site: one of the most important feature was to support faceted navigation on a set of hierarchical fields. After having done my research on search providers which currently exist in the market: Google search for enterprise, Mercado, Endeca, Omniture etc, we decided to go with solr (yes, the price was one of the major deciding factors: solr is free!) and I have to admit I haven't been disappointed with using it. Since, this is a windows/.net shop I to use solrsharp as the bridge. Installing Jetty+solr was a breeze but I ran into issues with installing Jetty as a service (so that we didn't have to run solr in console mode); also, I read somewhere that Jetty is not great in handling unicode characters (and since ours is a multi-lingual site; this was a big negative): so Jetty gave way to Tomcat; installing Tomcat as a service was easy on Win32 but I figured out that getting it to run on a 64bit Windows as a 64bit application required patching the Tomcat files from svn repo; after patching Tomcat with the required executables it installed just fine on a 64 bit m/c.
Since, solr supports faceted navigation out of the box (along with complex boolean queries), we had absolutely no issues in meeting the requirements; I did run into issues with solrsharp not handing unicode characters (had to patch it) and not supporting NOT searches (had to patch again) but once I patched solrsharp; things have been going on great. Currently, I use the Standard request handler (and it looks like solrsharp doesn't support specifying any different request handler during query time) which works fine for most of the cases but sometimes I've seen where my search results are quite off specially when you are searching on multiple terms for e.g. a search on "Joan of Arc" (w/o the quotes) should ideally return a document where Joan of Arc appear in close proximity first; than other documents; but the resultset returned by solr seem to be more based on words collision (the number of times a word appears in a doc). Also, if you're using the standard request handler; you can't assign different weights to different fields in the document so all fields are treated as same (you might want to assign more weight to Title field than description): the bad news is standard request handler can't handle all this complex stuff; the good news is there's another request handler just for this: Dismax. So, I just need to patch solrsharp so that I can tell it which request handler to use (or it can pick it from some config file) and we should be able to use Dismax as our request handler. All in all, it's been a breeze using solr: it's quite fast, supports clustering, runs just fine on win architecture and above all is free & open source!
Moral of the story: If you are planning to provide search functionality in your next site/application: use solr!