Cook Computing

Chunked Transfer and Aggregators

May 13, 2003 Written by Charles Cook

Trawling through my logs last night I came across a scenario which several of the more popular aggregators don't handle. If a feed is returned with the Transfer-Encoding header set to chunked and, presumably because content is being generated dynamically, for example through use of an SSI, no Last-Modified header, these aggregators don't bother sending an If-Modified-Since header. If the server does not support etags either, this results in the feed being downloaded on every request, instead of a 304 Not Modified response.

In this case the aggregator should use the time returned in the Date header as the value to be passed with If-Modified-Since in the next request. If the server supports If-Modified-Since, 304 will be returned unless the feed has changed.

This may be a fairly obscure secenario but other aggregators seem to handle it ok, so it is probably worth coding for. One caveat in this area: if a Last-Modified header was not returned, the LastModified property of the WebResponse class returns the local time on the client machine, not the value of the Date header on the HTTP response. If the clocks on the client and server machines are out of sync then things won't work quite as expected.