I'm not aware of Linux apps that are one-click and go on a USB stick, but that's a nice safety net to get to Ye 'olde Pogoplug when changing torrents and thereabouts and all you have is a Windows machine. You'd want these:
* WinSCP & PuTTY
* Firefox
Anything else i should add?
There's a large library of PortableApps, most as free, and most regular Windows apps are possible. Java's probably another one but i'm scared of it.