I use my DockStar mainly as a torrent seedbox under Debian Squeeze. Besides rtorrent, I also have sabnzbplus, samba, apache and other support aps for them. Recently, I add Asterisk 1.8 to DockStar so I can use GTalk for free calls within US (and Canada?). Most of the time, I think the unit can handles all the tasks, but I am not so sure during post processing by sabnzbplus to unrar/unzip whole bunch of files. Will it help if I use nice to up the priority of asterisk so it always has priority over others?
Thanks.

I would start by using the "top" program, which should allow you to see just how busy the processor is and how full your memory is. If you are seeing significant paging or swapping, you probably want to offload your real-time critical packages, like asterisk, to another Dockstar. You might be able to improve the situation by "nicing" the critical processes or locking them in memory, but paging and/or high CPU loads will in general kill real time app performance.

Yep, running top is the best way to monitor that. Change the ordering (Hit "O" and select which field it should sort by) to get an idea of what uses most CPU or RAM (since 128MB can be a little tight for some configs)