Welcome! Log In Create A New Profile

Advanced

Docker on Dell Wyse 3020 - need some help

Posted by chomick 
Forum List Message List New Topic
chomick
Docker on Dell Wyse 3020 - need some help
August 04, 2023 09:21AM
Hi guys!

So I have some commercial idea of usage of second-hand Dell 3020 devices but that would require running Docker on that device. As I mentioned in other topic I am computer literate, but really a newbie to linux systems and have barely any idea how it all works, but for past couple of weeks I've managed to learn some basics and I was able to boot Debian 11, ssh it, upgrade it to bookworm version and to install and run Docker, but I can't run any container in it.

So I 
systemctl start docker
then 
systemctl status docker

and I get 

● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; disabled; preset: enabled)
     Active: active (running) since Fri 2023-08-04 16:08:18 CEST; 23s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 1684 (dockerd)
      Tasks: 13
     Memory: 74.7M
        CPU: 3.940s
     CGroup: /system.slice/docker.service
             └─1684 /usr/sbin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Aug 04 16:08:11 debian dockerd[1684]: time="2023-08-04T16:08:11.110023233+02:00" level=info msg="[core] Subchannel Connectivity change to READY" module=grpc
Aug 04 16:08:11 debian dockerd[1684]: time="2023-08-04T16:08:11.110383781+02:00" level=info msg="[core] Channel Connectivity change to READY" module=grpc
Aug 04 16:08:11 debian dockerd[1684]: time="2023-08-04T16:08:11.485768230+02:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Aug 04 16:08:12 debian dockerd[1684]: time="2023-08-04T16:08:12.613523235+02:00" level=info msg="Loading containers: start."
Aug 04 16:08:16 debian dockerd[1684]: time="2023-08-04T16:08:16.644383747+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16.>
Aug 04 16:08:17 debian dockerd[1684]: time="2023-08-04T16:08:17.663586628+02:00" level=info msg="Loading containers: done."
Aug 04 16:08:18 debian dockerd[1684]: time="2023-08-04T16:08:18.381167821+02:00" level=info msg="Docker daemon" commit=5d6db84 graphdriver(s)=overlay2 version=20.10.24>
Aug 04 16:08:18 debian dockerd[1684]: time="2023-08-04T16:08:18.384701109+02:00" level=info msg="Daemon has completed initialization"
Aug 04 16:08:18 debian systemd[1]: Started docker.service - Docker Application Container Engine.
Aug 04 16:08:18 debian dockerd[1684]: time="2023-08-04T16:08:18.753309183+02:00" level=info msg="API listen on /run/docker.sock"

Then I 
docker run hello-world
and get 
[  965.645765][ T1687] docker0: port 1(veth0701d98) entered blocking state
[  965.683194][ T1687] docker0: port 1(veth0701d98) entered disabled state
[  965.700348][ T1687] veth0701d98: entered allmulticast mode
[  965.706507][ T1687] veth0701d98: entered promiscuous mode
[  968.481026][ T1687] docker0: port 1(veth0701d98) entered disabled state
[  968.492471][ T1687] veth0701d98 (unregistering): left allmulticast mode
[  968.500147][ T1687] veth0701d98 (unregistering): left promiscuous mode
[  968.506733][ T1687] docker0: port 1(veth0701d98) entered disabled state
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.
ERRO[0003] error waiting for container: context canceled

so I stop docker and docker.socket and run dockerd to listen. Then again, from other terminal I try to run hello-world. Dockerd displays: 
[ 1059.612140][ T1964] docker0: port 1(veth1f27828) entered blocking state
[ 1059.619560][ T1964] docker0: port 1(veth1f27828) entered disabled state
[ 1059.635206][ T1964] veth1f27828: entered allmulticast mode
[ 1059.641541][ T1964] veth1f27828: entered promiscuous mode
WARN[2023-08-04T16:12:34.264498301+02:00] seccomp is not enabled in your kernel, running container without default profile
ERRO[2023-08-04T16:12:35.034768348+02:00] stream copy error: reading from a closed fifo
ERRO[2023-08-04T16:12:35.045654363+02:00] stream copy error: reading from a closed fifo
[ 1061.729990][ T1966] docker0: port 1(veth1f27828) entered disabled state
[ 1061.743111][ T1966] veth1f27828 (unregistering): left allmulticast mode
[ 1061.754554][ T1966] veth1f27828 (unregistering): left promiscuous mode
[ 1061.761240][ T1966] docker0: port 1(veth1f27828) entered disabled state
ERRO[2023-08-04T16:12:36.620326408+02:00] 3c457a1d289fce7765dd052ff527798b7a485477835012da86c8281ae00b10f0 cleanup: failed to delete container from containerd: no such container
ERRO[2023-08-04T16:12:36.620782883+02:00] Handler for POST /v1.41/containers/3c457a1d289fce7765dd052ff527798b7a485477835012da86c8281ae00b10f0/start returned error: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown

Chatgpt has given me a hint, that whole this mess might be caused by lack of BPF, which stands for Berkeley Packet Filter

Whatsmore, I have found a script, which checks whether a device has all necesarry ingredients and configurations to run docker. Here's what I got: 


root@debian:~# ./check-config.sh
warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-6.3.8-mvebu-tld-1 ...

Generally Necessary:
- cgroup hierarchy: cgroupv2
  Controllers:
  - cpu: available
  - cpuset: available
  - io: available
  - memory: available
  - pids: available
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: missing

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: missing
- CONFIG_SECCOMP_FILTER: missing
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT3_FS: enabled
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: enabled
- CONFIG_EXT3_FS_SECURITY: enabled
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
    - CONFIG_BRIDGE_VLAN_FILTERING: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_NETFILTER_XT_MATCH_BPF: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

My conclusion would be, that the (only) thing that stops me is a lack of BPF module.

Would it be possible to add it to the kernel? Would I be able to do it by myself, or does it require real profficiency? Or are there any other limitations resulting from what I've presented, or any other source?
Reply Quote
bodhi
Re: Docker on Dell Wyse 3020 - need some help
August 04, 2023 09:46AM
chomick Wrote:
-------------------------------------------------------
> Hi guys!
>
> So I have some commercial idea of usage of
> second-hand Dell 3020 devices but that would
> require running Docker on that device. As I
> mentioned in other topic I am computer literate,
> but really a newbie to linux systems and have
> barely any idea how it all works, but for past
> couple of weeks I've managed to learn some basics
> and I was able to boot Debian 11, ssh it, upgrade
> it to bookworm version and to install and run
> Docker, but I can't run any container in it.
>
> So I
> 
> systemctl start docker
>
> then
> 
> systemctl status docker
>
>
> and I get
>
> 
> ● docker.service - Docker Application Container
> Engine
>      Loaded: loaded
> (/lib/systemd/system/docker.service; disabled;
> preset: enabled)
>      Active: active (running) since Fri 2023-08-04
> 16:08:18 CEST; 23s ago
> TriggeredBy: ● docker.socket
>        Docs: https://docs.docker.com
>    Main PID: 1684 (dockerd)
>       Tasks: 13
>      Memory: 74.7M
>         CPU: 3.940s
>      CGroup: /system.slice/docker.service
>              └─1684 /usr/sbin/dockerd -H fd://
> --containerd=/run/containerd/containerd.sock
> 
> Aug 04 16:08:11 debian dockerd[1684]:
> time="2023-08-04T16:08:11.110023233+02:00"
> level=info msg="[core] Subchannel Connectivity
> change to READY" module=grpc
> Aug 04 16:08:11 debian dockerd[1684]:
> time="2023-08-04T16:08:11.110383781+02:00"
> level=info msg="[core] Channel Connectivity change
> to READY" module=grpc
> Aug 04 16:08:11 debian dockerd[1684]:
> time="2023-08-04T16:08:11.485768230+02:00"
> level=info msg="[graphdriver] using prior storage
> driver: overlay2"
> Aug 04 16:08:12 debian dockerd[1684]:
> time="2023-08-04T16:08:12.613523235+02:00"
> level=info msg="Loading containers: start."
> Aug 04 16:08:16 debian dockerd[1684]:
> time="2023-08-04T16:08:16.644383747+02:00"
> level=info msg="Default bridge (docker0) is
> assigned with an IP address 172.17.0.0/16.>
> Aug 04 16:08:17 debian dockerd[1684]:
> time="2023-08-04T16:08:17.663586628+02:00"
> level=info msg="Loading containers: done."
> Aug 04 16:08:18 debian dockerd[1684]:
> time="2023-08-04T16:08:18.381167821+02:00"
> level=info msg="Docker daemon" commit=5d6db84
> graphdriver(s)=overlay2 version=20.10.24>
> Aug 04 16:08:18 debian dockerd[1684]:
> time="2023-08-04T16:08:18.384701109+02:00"
> level=info msg="Daemon has completed
> initialization"
> Aug 04 16:08:18 debian systemd[1]: Started
> docker.service - Docker Application Container
> Engine.
> Aug 04 16:08:18 debian dockerd[1684]:
> time="2023-08-04T16:08:18.753309183+02:00"
> level=info msg="API listen on /run/docker.sock"
> 
>
>
> Then I
> 
> docker run hello-world
>
> and get
> 
> [  965.645765][ T1687] docker0: port
> 1(veth0701d98) entered blocking state
> [  965.683194][ T1687] docker0: port
> 1(veth0701d98) entered disabled state
> [  965.700348][ T1687] veth0701d98: entered
> allmulticast mode
> [  965.706507][ T1687] veth0701d98: entered
> promiscuous mode
> [  968.481026][ T1687] docker0: port
> 1(veth0701d98) entered disabled state
> [  968.492471][ T1687] veth0701d98
> (unregistering): left allmulticast mode
> [  968.500147][ T1687] veth0701d98
> (unregistering): left promiscuous mode
> [  968.506733][ T1687] docker0: port
> 1(veth0701d98) entered disabled state
> docker: Error response from daemon: failed to
> create shim task: OCI runtime create failed: runc
> create failed: unable to start container process:
> error during container init: error setting cgroup
> config for procHooks process:
> bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid
> argument: unknown.
> ERRO[0003] error waiting for container: context
> canceled
>
>
> so I stop docker and docker.socket and run dockerd
> to listen. Then again, from other terminal I try
> to run hello-world. Dockerd displays:
> 
> [ 1059.612140][ T1964] docker0: port
> 1(veth1f27828) entered blocking state
> [ 1059.619560][ T1964] docker0: port
> 1(veth1f27828) entered disabled state
> [ 1059.635206][ T1964] veth1f27828: entered
> allmulticast mode
> [ 1059.641541][ T1964] veth1f27828: entered
> promiscuous mode
> WARN[2023-08-04T16:12:34.264498301+02:00] seccomp
> is not enabled in your kernel, running container
> without default profile
> ERRO[2023-08-04T16:12:35.034768348+02:00] stream
> copy error: reading from a closed fifo
> ERRO[2023-08-04T16:12:35.045654363+02:00] stream
> copy error: reading from a closed fifo
> [ 1061.729990][ T1966] docker0: port
> 1(veth1f27828) entered disabled state
> [ 1061.743111][ T1966] veth1f27828
> (unregistering): left allmulticast mode
> [ 1061.754554][ T1966] veth1f27828
> (unregistering): left promiscuous mode
> [ 1061.761240][ T1966] docker0: port
> 1(veth1f27828) entered disabled state
> ERRO[2023-08-04T16:12:36.620326408+02:00]
> 3c457a1d289fce7765dd052ff527798b7a485477835012da86c8281ae00b10f0
> cleanup: failed to delete container from
> containerd: no such container
> ERRO[2023-08-04T16:12:36.620782883+02:00] Handler
> for POST
> /v1.41/containers/3c457a1d289fce7765dd052ff527798b7a485477835012da86c8281ae00b10f0/start
> returned error: failed to create shim task: OCI
> runtime create failed: runc create failed: unable
> to start container process: error during container
> init: error setting cgroup config for procHooks
> process: bpf_prog_query(BPF_CGROUP_DEVICE) failed:
> invalid argument: unknown
>
>
> Chatgpt has given me a hint, that whole this mess
> might be caused by lack of BPF, which
> stands for Berkeley Packet Filter
>
> Whatsmore, I have found a script, which checks
> whether a device has all necesarry ingredients and
> configurations to run docker. Here's what I got:
>
> 
> 
> root@debian:~# ./check-config.sh
> warning: /proc/config.gz does not exist, searching
> other paths for kernel config ...
> info: reading kernel config from
> /boot/config-6.3.8-mvebu-tld-1 ...
> 
> Generally Necessary:
> - cgroup hierarchy: cgroupv2
>   Controllers:
>   - cpu: available
>   - cpuset: available
>   - io: available
>   - memory: available
>   - pids: available
> - CONFIG_NAMESPACES: enabled
> - CONFIG_NET_NS: enabled
> - CONFIG_PID_NS: enabled
> - CONFIG_IPC_NS: enabled
> - CONFIG_UTS_NS: enabled
> - CONFIG_CGROUPS: enabled
> - CONFIG_CGROUP_CPUACCT: enabled
> - CONFIG_CGROUP_DEVICE: enabled
> - CONFIG_CGROUP_FREEZER: enabled
> - CONFIG_CGROUP_SCHED: enabled
> - CONFIG_CPUSETS: enabled
> - CONFIG_MEMCG: enabled
> - CONFIG_KEYS: enabled
> - CONFIG_VETH: enabled (as module)
> - CONFIG_BRIDGE: enabled (as module)
> - CONFIG_BRIDGE_NETFILTER: enabled (as module)
> - CONFIG_IP_NF_FILTER: enabled (as module)
> - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as
> module)
> - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as
> module)
> - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as
> module)
> - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as
> module)
> - CONFIG_NETFILTER_XT_MARK: enabled (as module)
> - CONFIG_IP_NF_NAT: enabled (as module)
> - CONFIG_NF_NAT: enabled (as module)
> - CONFIG_POSIX_MQUEUE: enabled
> - CONFIG_CGROUP_BPF: missing
> 
> Optional Features:
> - CONFIG_USER_NS: enabled
> - CONFIG_SECCOMP: missing
> - CONFIG_SECCOMP_FILTER: missing
> - CONFIG_CGROUP_PIDS: enabled
> - CONFIG_MEMCG_SWAP: missing
>     (cgroup swap accounting is currently enabled)
> - CONFIG_BLK_CGROUP: enabled
> - CONFIG_BLK_DEV_THROTTLING: missing
> - CONFIG_CGROUP_PERF: enabled
> - CONFIG_CGROUP_HUGETLB: missing
> - CONFIG_NET_CLS_CGROUP: enabled
> - CONFIG_CGROUP_NET_PRIO: missing
> - CONFIG_CFS_BANDWIDTH: missing
> - CONFIG_FAIR_GROUP_SCHED: enabled
> - CONFIG_IP_NF_TARGET_REDIRECT: enabled (as
> module)
> - CONFIG_IP_VS: enabled (as module)
> - CONFIG_IP_VS_NFCT: enabled
> - CONFIG_IP_VS_PROTO_TCP: enabled
> - CONFIG_IP_VS_PROTO_UDP: enabled
> - CONFIG_IP_VS_RR: enabled (as module)
> - CONFIG_SECURITY_SELINUX: enabled
> - CONFIG_SECURITY_APPARMOR: missing
> - CONFIG_EXT3_FS: enabled
> - CONFIG_EXT3_FS_XATTR: missing
> - CONFIG_EXT3_FS_POSIX_ACL: enabled
> - CONFIG_EXT3_FS_SECURITY: enabled
>     (enable these ext3 configs if you are using
> ext3 as backing filesystem)
> - CONFIG_EXT4_FS: enabled
> - CONFIG_EXT4_FS_POSIX_ACL: enabled
> - CONFIG_EXT4_FS_SECURITY: enabled
> - Network Drivers:
>   - "overlay":
>     - CONFIG_VXLAN: missing
>     - CONFIG_BRIDGE_VLAN_FILTERING: missing
>       Optional (for encrypted networks):
>       - CONFIG_CRYPTO: enabled
>       - CONFIG_CRYPTO_AEAD: enabled
>       - CONFIG_CRYPTO_GCM: enabled
>       - CONFIG_CRYPTO_SEQIV: enabled
>       - CONFIG_CRYPTO_GHASH: enabled
>       - CONFIG_XFRM: enabled
>       - CONFIG_XFRM_USER: enabled (as module)
>       - CONFIG_XFRM_ALGO: enabled (as module)
>       - CONFIG_INET_ESP: enabled (as module)
>       - CONFIG_NETFILTER_XT_MATCH_BPF: enabled (as
> module)
>   - "ipvlan":
>     - CONFIG_IPVLAN: enabled (as module)
>   - "macvlan":
>     - CONFIG_MACVLAN: enabled (as module)
>     - CONFIG_DUMMY: enabled (as module)
>   - "ftp,tftp client in container":
>     - CONFIG_NF_NAT_FTP: enabled (as module)
>     - CONFIG_NF_CONNTRACK_FTP: enabled (as
> module)
>     - CONFIG_NF_NAT_TFTP: enabled (as module)
>     - CONFIG_NF_CONNTRACK_TFTP: enabled (as
> module)
> - Storage Drivers:
>   - "btrfs":
>     - CONFIG_BTRFS_FS: enabled (as module)
>     - CONFIG_BTRFS_FS_POSIX_ACL: enabled
>   - "overlay":
>     - CONFIG_OVERLAY_FS: enabled
>   - "zfs":
>     - /dev/zfs: missing
>     - zfs command: missing
>     - zpool command: missing
> 
> Limits:
> - /proc/sys/kernel/keys/root_maxkeys: 1000000
>
>
> My conclusion would be, that the (only) thing that
> stops me is a lack of BPF module.
>
> Would it be possible to add it to the kernel?
> Would I be able to do it by myself, or does it
> require real profficiency? Or are there any other
> limitations resulting from what I've presented, or
> any other source?

Yes, you are on the right track! Let me doublecheck my notes, but I recall that I need to add a few kernel configs to support more BPF and also SECCOMP to support docker. Will do that in the next kernel release.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 08/04/2023 09:54AM by bodhi.
Reply Quote
chomick
Re: Docker on Dell Wyse 3020 - need some help
August 05, 2023 10:41AM
> I recall that I need to
> add a few kernel configs to support more BPF and
> also SECCOMP to support docker. Will do that in
> the next kernel release.

so I know that u will do that sooner that I will learn how to do it, but if u could please tell - how do you reckon, when next tarball will be ready?
Reply Quote
bodhi
Re: Docker on Dell Wyse 3020 - need some help
August 05, 2023 02:04PM
> so I know that u will do that sooner that I will
> learn how to do it, but if u could please tell -
> how do you reckon, when next tarball will be
> ready?

I don't have any fixed schedule. Usually, I release a new kernel every 2 months, during the later part of a new minor kernel version, i.e. 6.4.x. I'm still on travel so it probably will be sometime at the end of this month.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 08/06/2023 03:54AM by bodhi.
Reply Quote
Newer Topic Older Topic
Author:

Your Email:


Subject:
Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: