Brandon, I don't think it is the Dockstar hardware that's the bottleneck, at least if all you are trying to do is stream programming without conversion. Outwardly, my setup is quite different from yours, but I am doing something similar. I don't have a Bluetooth player, PS3, nor mediatomb, and my WiFi router is located several switches away from the Dockstar. However, I've been using the Dockstar and its big-brother-Plugbox, the SheevaPlug, to record and watch OTA 1024i programming for many months now with no real problems. I pull in the programming from an HDHomeRun tuner and save it on the Dockstar's HD. I can then export it and watch it over my wired 100baseT LAN, without problems, using VLC on a remote Linux box or W7 laptop. Indeed, I can watch a 1080i program while recording another two with nary a hiccup, and stream to two devices simultaneously.
Here's what I've concluded after many months of doing this:
The Marvel-based computers are perfectly capable of streaming one or two 1080i programs into and/or out of the device simultaneously at 100BaseT speeds without problems. Thus, I would think they should handle one 1080p stream.
Both NFS and Samba are capable of serving up this programming. I think NFS has less overhead. I have not tried streaming multiple programs simultaneously via Samba.
My LAN is configured in a star pattern, with switches in every room all tied to a common switch in the basement. Thus, going room-to-room involves three switches. Traversing several switches is frankly a problem. I see dropped packets, although I don't attribute this to the Dockstar.
Likewise, if I try to use WiFi, the results are borderline. WiFi in conjunction with several hardwired switches simply doesn't work. A Netgear WG602 Access Point on the same switch works better, but I still occasionally see dropped frames. Sometimes it's watchable, sometimes not.
Anyway, that's the data points I can present. Good luck with interacting with your PS3. If you get it working reliably, let us know what you did.