I've always carried a small backpack with me pretty much everywhere I go. This is perhaps a hold-over from school and uni, continued because I always want to have a couple of books on me. These days, however, there's a lot of other stuff in there as well.
First of all, cycling around Oxford in winter means I'm carrying:
* a scarf
* a beanie
* a bright yellow cycling shell - windproof and sort of waterproof
* nylon overpants (when rain seems likely)
* a bicycle seat cover (when rain seems likely)
* front and back bike lights
That still leaves room for a couple of books, pens, boxes of tissues, and so forth, along with some spare bags for shopping. I usually do larger grocery runs on foot, so I can carry a couple of separate shopping bags, but I should perhaps get a set of pannier bags for that.