Syncthing on GoFlex Home

Hi!
I understand that I'm probably stretching this box a bit too hard, but I'm trying to run a Samba server, Syncthing and also Transmission simultaneously on my little GoFlex Home box.
The scenario is not as extreme as it may seem, because Transmission is idle most of the time, with no active torrents, and the same thing happens with Syncthing and Samba, which are not consciously put to use by me at the same time.

The problem I find is that the Syncthing services terminate unexpectedly at strange times (e.g. during the night, with no use) and I can't find nothing in the logs (logread, smesg, syncthing own panic logs) that is helpful (to me).

The only clues I have are probably related to low memory issues:
- Syncthing logs sometimes show some SIGSEGV error, which appears that might be related to low memory issues or application bugs
- I think that even before installing Syncthing, and perhaps after some kernel or debian upgrade (I'm not really sure), also smdb got somehow terminated and even apt (or apt-get) refused to run, issues that could be solved by rebooting or just by "dropping caches" (e.g., echo 3 > /proc/sys/vm/drop_caches)

I have even tried to follow the "Tuning for low RAM boxes" topic in the Wiki, but noticed no differences.

Has anyone experienced something similar and has some tips to share? Thanks!

Nuno,

Quote

- I think that even before installing Syncthing, and perhaps after some kernel or debian upgrade (I'm not really sure), also smdb got somehow terminated and even apt (or apt-get) refused to run, issues that could be solved by rebooting or just by "dropping caches" (e.g., echo 3 > /proc/sys/vm/drop_caches)

I have even tried to follow the "Tuning for low RAM boxes" topic in the Wiki, but noticed no differences.

That should take care of transmission and samba.

Quote

- Syncthing logs sometimes show some SIGSEGV error, which appears that might be related to low memory issues or application bugs

1. Is there memory leak? (by Syncthing)
2. If there is no memory leak then it sounds like bug in the daemon. With swap space >= 4 times RAM, the worst can happen in consistently low memeory situation is it will cause disk thrashing.

-bodhi
===========================
Forum Wiki
bodhi's corner

Thanks for your reply!
I am not really sure if I'll be able to find if there's a memory leak or not in Syncthing (I'm not really a developer). I did search for possible bugs and other issues in Syncthing's forum and github issues section, but coulnd't find any. I'm guessing there's not many people with a setup like mine (I do enjoy stretching these little boxes a bit!).

I took a quick look using htop, sorted by MEM% and Syncthing takes all the first places! I guess that's to be expected, but that 795/794M in the VIRT column seems (to me) a bit puzzling... Perhaps someone can tell me if this is to be expected or if I should lower my expectations... :-)

  CPU[|||||                     9.2%]   Tasks: 27, 20 thr; 1 running
  Mem[||||||||||||||||||||52.6M/108M]   Load average: 0.23 0.81 0.61
  Swp[|||                 34.8M/512M]   Uptime: 19:52:13

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%▽  TIME+  Command
 3666 root       31  11  795M 18992  7912 S  5.9 17.2  1:13.77 /usr/bin/syncthin
 3667 root       31  11  795M 18992  7912 S  0.0 17.2  0:05.75 /usr/bin/syncthin
 3668 root       31  11  795M 18992  7912 S  0.0 17.2  0:00.02 /usr/bin/syncthin
 3669 root       31  11  795M 18992  7912 S  0.0 17.2  0:00.00 /usr/bin/syncthin
 3670 root       31  11  795M 18992  7912 S  0.0 17.2  0:00.00 /usr/bin/syncthin
 3671 root       31  11  795M 18992  7912 S  0.0 17.2  0:16.84 /usr/bin/syncthin
 3672 root       31  11  795M 18992  7912 S  0.0 17.2  0:19.18 /usr/bin/syncthin
 3673 root       31  11  795M 18992  7912 S  5.3 17.2  0:17.43 /usr/bin/syncthin
 3674 root       31  11  795M 18992  7912 S  0.0 17.2  0:14.17 /usr/bin/syncthin
 3644 root       20   0  794M  2312   364 S  0.0  2.1  0:01.99 /usr/bin/syncthin
 3650 root       20   0  794M  2312   364 S  0.0  2.1  0:00.32 /usr/bin/syncthin
 3651 root       20   0  794M  2312   364 S  0.0  2.1  0:00.25 /usr/bin/syncthin
 3652 root       20   0  794M  2312   364 S  0.0  2.1  0:00.00 /usr/bin/syncthin
 3653 root       20   0  794M  2312   364 S  0.0  2.1  0:00.00 /usr/bin/syncthin
 3654 root       20   0  794M  2312   364 S  0.0  2.1  0:00.05 /usr/bin/syncthin
 3665 root       20   0  794M  2312   364 S  0.0  2.1  0:01.21 /usr/bin/syncthin
 3684 root       20   0  8708  1564  1064 R  2.0  1.4  0:02.29 htop

VIRT is virtual memory, in this case that is basically only address space which is reserved. That address space does not have to be shared with other processes, as each process has it's own 4GiB of potential address space.
RES (resident) is the actual amount of used memory (+ the size of the binaries in memory).

You can detect a memory leak be watching the memory usage over time. If it keeps growing, then you have a leak. The program can crash when it's out of virtual address space (2 or 3 GiB, depending on kernel config), or it can be killed when the system runs out of memory. In the latter case the OOM killer kicks in, and that should be logged in the kernel log. Crashing due to out of address space is probably nowhere logged.

Seeing this htop dump it doesn't seem to me your expectations are too high. Half the memory is still available. Even if you omit the swapfile it should still fit.

If syncthing has a leak, then you can (apart from asking help from the syncthing developers) pre-emptive restart it each hour, or whatever amount of time the program needs to get too big.

Quote

Seeing this htop dump it doesn't seem to me your expectations are too high. Half the memory is still available. Even if you omit the swapfile it should still fit.

+1

The OOM killer is not likely to kick in with this type of memory usage (if there is no sudden memory leak).

Also the kernel is configured with CONFIG_IOSCHED_BFQ. So I would expect the disk thrashing to occur first when the system consumes lots of memory. Perhaps the OOM killer will never kick in, in a normal case with no leak.

-bodhi
===========================
Forum Wiki
bodhi's corner

@Mihzelf
@bodhi

Thank you both for your answers. It's nice to know that my expectations seem to be reasonable.
I'll try to keep an eye on Syncthing's memory usage and try to find if the crash is being caused by something other than a memory issue.

I'd like to just to let you know I might have found the issue.
After playing around with f3write/f3read I found out that the SD card I've been using with my GoFlex is a bit unreliable (not fake, just with some random errors). The surprise is that every other SD I tried (some 3 others, with 4 different USB adapters) behaved nearly the same.
I finally managed to find a regular USB flash drive that has performed flawlessly since, albeit a bit slow.
Using debsums, I also found a few corrupted files and reinstalled the corresponding packages (I had cloned the previous SD to the new flash drive).
Since performing this verification/repair, my GoFlex Home has been perfectly fine, with no downtime, and Syncthing hasn't closed unexpectedly even once.