Category Archives: tech

CentOS 8 + AqBanking + HBCI + Commerzbank

As of 01/2021, this is a working method:

dnf install epel-release
dnf install dnf-plugins-core
dnf config-manager –set-enabled PowerTools
yum install aqbanking
(This will install AqBanking 6.1.4, which is recent enough)

gct-tool create -t ohbci -n cb.medium.2021

aqhbci-tool4 adduser -t ohbci -n cb.medium.2021 –context=1 -b 50040000 -u TEILNEHMERNUMMER -c TEILNEHMERNUMMER -s hbci.commerzbank.de -N SomeIdentifierDoesntMatter –rdhtype=10 –cryptmoderah –hbciversion=300

aqhbci-tool4 getkeys -u 1
aqhbci-tool4 createkeys -u 1
aqhbci-tool4 sendkeys -A -u 1
aqhbci-tool4 iniletter -u 1

At this point, send INI letter to your Commerzbank guy for activation. After activation check that everything works as expected:

aqhbci-tool4 getsysid -u 1
aqhbci-tool4 getaccounts -u 1
aqbanking-cli listaccs -b 50040000
aqbanking-cli request –transactions -b 50040000 -c test.txt
aqbanking-cli export –exporter=csv -c test.txt -o transactions.csv

galera: Won’t start if there’s only one node

Leave a reply

Stupid Galera fucks up every now and then. If there’s only one node left and it doesn’t start with:

[ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)

Then you need to “bootstrap” it:

In grastate.dat:

safe_to_bootstrap: 0

Must be 1 not 0!

And then just do:

service mysql start –wsrep-new-cluster

Unstable crap…

CentOS 6: Switching from mod_php to fastcgi (Link only)

Leave a reply

This short tutorial for Apache 2.2 worked:

https://www.kutukupret.com/2016/06/29/centos-6-httpd-2-2-and-php-fpm/

Actually, the following should have worked too, there’s no mention for the need of installing fastcgi at all, I assumed php-fpm by itself would be enough but that doesn’t seem to be the case. Maybe an Apache 2.4 thing?

https://developers.redhat.com/blog/2017/10/25/php-configuration-tips/

Finding the 100% CPU culprit in multi-threaded applications

Leave a reply

Wow, after many years with Linux I just stubled upon some ultra useful functionality and now I feel kind of stupid because I didn’t know about it all that time, lol. :)

I have a multi-threaded application – namely Asterisk, a software PBX – that was always at 250% CPU for many months without a visible reason. Googling for Asterisk + high CPU brought me to this site: https://moythreads.com/wordpress/2009/05/06/why-does-asterisk-consume-100-cpu

Basically everything is already explained there but I’ll give another example. Here sample output from top:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16780 asterisk -11   0 4681m 115m  10m S 207.5  3.0 649:34.98 asterisk

So, Asterisk going berserk at 207% CPU. Let’s install pstack:

yum install gdb

Let’s look at all the threads of Asterisk, PID 16780, using the magic parameters -LlFm to ps which will show all threads of that process:

ps -LlFm 16780

The output will be something like the following. Look for the C column which means CPU usage:

F S UID        PID  PPID   LWP  C NLWP PRI  NI ADDR SZ WCHAN    RSS PSR STIME TTY        TIME CMD
[...]
1 S asterisk     -     - 32498  0    -  49   - -     - futex_     -   7 04:15 -        00:00:00 -
1 S asterisk     -     - 32499  0    -  49   - -     - poll_s     -   1 04:15 -        00:00:00 -
1 S asterisk     -     -   418  0    -  49   - -     - inotif     -   4 04:33 -        00:00:00 -
1 R asterisk     -     -  3967 99    -  49   - -     - -          -   0 05:59 -        12:14:44 -
1 R asterisk     -     -  4367 99    -  49   - -     - -          -   1 06:05 -        12:08:27 -
1 S asterisk     -     - 22668  0    -  49   - -     - poll_s     -   4 16:40 -        00:00:19 -
1 S asterisk     -     - 23627  0    -  49   - -     - poll_s     -   7 17:20 -        00:00:12 -
1 S asterisk     -     - 23641  0    -  49   - -     - poll_s     -   2 17:20 -        00:00:11 -
[...]

Notice those two entries with 99 CPU: LWP 3967 and 4367! Let’s look at these little f*ckers in more detail:

pstack 16780 > /tmp/asterisk.stack.txt

Let’s look into /tmp/asterisk.stack.txt and search for our two LWPs 3967 and 4367…

Thread 35 (Thread 0x7ff211a9b700 (LWP 3967)):
#0  0x00007ff248cb96ec in recv () from /lib64/libc.so.6
#1  0x00007ff23c07b8b1 in ooSocketRecv () from /usr/lib64/asterisk/modules/chan_ooh323.so
#2  0x00007ff23c06461f in ooH2250Receive () from /usr/lib64/asterisk/modules/chan_ooh323.so
#3  0x00007ff23c064fba in ooProcessCallFDSETsAndTimers () from /usr/lib64/asterisk/modules/chan_ooh323.so
#4  0x00007ff23c06518e in ooMonitorCallChannels () from /usr/lib64/asterisk/modules/chan_ooh323.so
#5  0x00007ff23c14ed95 in ooh323c_call_thread () from /usr/lib64/asterisk/modules/chan_ooh323.so
#6  0x000000000057a1a8 in dummy_start ()
#7  0x00007ff2476eaaa1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007ff248cb893d in clone () from /lib64/libc.so.6
Thread 34 (Thread 0x7ff21195e700 (LWP 4367)):
#0  0x00007ff248cb96ec in recv () from /lib64/libc.so.6
#1  0x00007ff23c07b8b1 in ooSocketRecv () from /usr/lib64/asterisk/modules/chan_ooh323.so
#2  0x00007ff23c06461f in ooH2250Receive () from /usr/lib64/asterisk/modules/chan_ooh323.so
#3  0x00007ff23c064fba in ooProcessCallFDSETsAndTimers () from /usr/lib64/asterisk/modules/chan_ooh323.so
#4  0x00007ff23c06518e in ooMonitorCallChannels () from /usr/lib64/asterisk/modules/chan_ooh323.so
#5  0x00007ff23c14ed95 in ooh323c_call_thread () from /usr/lib64/asterisk/modules/chan_ooh323.so
#6  0x000000000057a1a8 in dummy_start ()
#7  0x00007ff2476eaaa1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007ff248cb893d in clone () from /lib64/libc.so.6

Aha! Something about ooh323 which is a module responsible for H.323 which we really don’t need in 2019. Let’s deactivate the culprit and put this into /etc/asterisk/modules.conf:

noload => chan_ooh323.so

Restart asterisk and voila:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25893 asterisk -11   0 4601m  64m  10m S 14.3  1.7   3:16.92 asterisk

14% CPU is about what Asterisk really consumes given the load. Problem solved. pstack and ps to the rescue!

galera: State not recoverable

Leave a reply

Once a year or so Galera on one of my 3 nodes breaks down and in the log you can find something like:

Failed to open channel ... State not recoverable

In my case, I end up with a empty gvwstate.dat file and that’s the problem. I delete the file and restart MySQL, then Galera syncs with the other nodes and everything is fine again.

Link to another article: https://github.com/codership/galera/issues/354

No MySQL root access when installing Galera from binaries on CentOS 7

Leave a reply

EDIT: I AM the one who is stupid! Arrrrgh. Just installed Galera on yet another box and saw this passing by in yum:

A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER ! You will find that password in '/root/.mysql_secret'.

Arrrrghhhh. Apologies to the RPM packagers. EDIT END.

Stupid RPM packagers screwed up. If you follow this how-to – http://galeracluster.com/documentation-webpages/gettingstarted.html – after a while you end up at a point where it says “In the database client, run the following query: SHOW STATUS LIKE 'wsrep_cluster_size'; …so you try “mysql -p” only to find that you don’t have access. WTF. There’s probably already a password set by the RPM packagers but we don’t know it. So, we try our usual –skip-grant-tables thing and then try SET PASSWORD FOR 'root'@'localhost' = PASSWORD('MyNewPass'); but this will result in: “ERROR 1290 (HY000): The MySQL server is running with the --skip-grant-tables option so it cannot execute this statement” – WTF again. When you are used to resetting MySQL root pw’s you usually first run the PW change command (SET or ALTER or UPDATE or whatever) and then you enter FLUSH PRIVILEGES. The trick here is to do the opposite. First enter

mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec)

mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('MyNewPass'); Query OK, 0 rows affected (0.00 sec)

Found this here after hours of troubleshooting: http://galeracluster.com/community/?place=msg%2Fcodership-team%2Fw8NEekKipwY%2FgGlkSQNOedMJ
F*CKERS!

Quick copy & paste how-to: Install KVM on CentOS 7

Leave a reply

CentOS 7.2.1511 64bit ftp://ftp.hosteurope.de/mirror/centos.org/7.2.1511/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso


URL for netinstall: ftp://ftp.hosteurope.de/mirror/centos.org/7/os/x86_64
Host: LV root: 200 GB
(Somehow, VG group size was not being offered during install, so after install: Removed partition with fdisk, create

new one, fdisk knows start/end cylinder by itself, choose Linux LVM partition type: 8e, write, reboot,

pvresize /dev/sda2, done)
vi /etc/sysconfig/selinux

SELINUX=disabled
yum install wget screen lynx vim libvirt qemu-kvm tigervnc tigervnc-server system-config-firewall-tui xterm bridge-utils virt-manager openssh-askpass
yum groupinstall "GNOME Desktop"
yum update
vi /usr/lib/systemd/system/vncserver@.service

replace USER with root
vi /etc/ssh/sshd_config

Port 450

PermitRootLogin without-password
adduser admin

passwd admin
systemctl disable firewalld

systemctl disable wpa_supplicant

systemctl disable bluetooth

systemctl disable alsa-state

systemctl disable abrtd

systemctl disable ModemManager

systemctl disable NetworkManager

systemctl disable cups

systemctl disable avahi-daemon

systemctl disable gssproxy

systemctl disable dnsmasq

systemctl disable upower

systemctl disable udisks2

systemctl disable colord

systemctl disable chronyd

systemctl disable auditd

systemctl disable libstoragemgmt

systemctl disable accounts-daemon

systemctl disable packagekit

systemctl disable polkit
vi /etc/sysconfig/network-scripts/ifcfg-enp3s0

TYPE="Ethernet"

BOOTPROTO="none"

DEFROUTE="yes"

IPV4_FAILURE_FATAL="no"

IPV6INIT="no"

IPV6_AUTOCONF="yes"

IPV6_DEFROUTE="yes"

IPV6_PEERDNS="yes"

IPV6_PEERROUTES="yes"

IPV6_FAILURE_FATAL="no"

NAME="enp3s0"

UUID="ac5861af-de34-416d-afcc-915776a8e4f0"

DEVICE="enp3s0"

ONBOOT="yes"

# IPADDR="x.x.x.x"

PREFIX="24"

# GATEWAY="x.x.x.x"

DNS1="8.8.8.8"

DNS2="8.8.4.4"

DNS3="1.2.3.4"

DOMAIN="something.net"

USERCTL=no

BRIDGE=br0
vi /etc/sysconfig/network-scripts/ifcfg-br0

DEVICE=br0

TYPE=Bridge

BOOTPROTO=none

ONBOOT=yes

DELAY=0

NM_CONTROLLED=no

BROADCAST=x.x.x.x

GATEWAY=x.x.x.x

IPADDR=x.x.x.x

NETMASK=x.x.x.x

DNS1=x.x.x.x

DNS2=x.x.x.x

DNS3=x.x.x.x

IPV6INIT=no

USERCTL=no
vi /etc/sysctl.conf

net.ipv4.ip_forward = 1

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1
vi /etc/postfix/main.cf:

inet_protocols = ipv4
Add hostname to /etc/hosts
reboot

vncpasswd systemctl start vncserver@:10.service

VNC connect to host:10

virt-manager:

QEMU/KVM ->

1. Virtual Networks -> default -> (Device: virbr0) -> Autostart: unchecked (Never) -> Apply

2. Storage -> + (add) -> Name: “vm-lvm” -> Type: logical: LVM Volume Group -> > Target Path: /dev/ber06 (enter path to VG group manually :-( ) -> Build Pool: unchecked -> Finish

OpenNebula & CentOS: OneFlow doesn’t start

Leave a reply

If OneFlow doesn’t start and you find this in your /var/log/one/oneflow.error:

/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- treetop (LoadError) from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' from /usr/lib/one/oneflow/lib/models/role.rb:17 from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' from /usr/lib/one/oneflow/lib/models.rb:26 from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' from /usr/lib/one/oneflow/oneflow-server.rb:49

Then do:

gem install treetop polyglot parse-cron

Although treetop is in EPEL (as rubygem-treetop.noarch) you get this when you try to install it:

Error: Package: rubygem-treetop-1.4.10-1.el6.noarch (epel) Requires: rubygem(polyglot)

It depends on polyglot but polyglot is not in EPEL – how stupid is that? And nowhere else to be found. Some other 3rd party repos claim to have it but adding another repo just for a single ruby gem – no thanks.

And you also need parse-cron, sigh.

CentOS: Fix broken yum repo metadata

Leave a reply

Happened because I added EPEL, I believe. Very strange, on another, identical machine (steps 1:1) it worked fine, but on this box suddenly this appeared when trying to yum install something:

[...] --> Processing Dependency: libnuma.so.1()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64 --> Processing Dependency: libnl.so.1()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64 --> Processing Dependency: libnetcf.so.1()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64 --> Processing Dependency: libgnutls.so.26()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64 --> Processing Dependency: libavahi-common.so.3()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64 --> Processing Dependency: libavahi-client.so.3()(64bit) for package: libvirt-0.10.2-29.el6_5.11.x86_64 ---> Package qemu-kvm.x86_64 2:0.12.1.2-2.415.el6_5.10 will be installed http://mirror2.hs-esslingen.de/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. http://centos.mirror.sharkservers.co.uk/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. http://ftp.plusline.de/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. http://ftp.hosteurope.de/mirror/centos.org/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. http://mirror.netcologne.de/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. http://centos.intergenia.de/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. http://centos.bio.lmu.de/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. http://centos.psw.net/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. http://mirror.maeh.org/centos/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. ftp://mirror.fraunhofer.de/centos.org/6.5/updates/x86_64/repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2: [Errno 14] PYCURL ERROR 19 - "Given file does not exist" Trying other mirror. Error: failure: repodata/607e7e1f0586f3b6c3478b8b07debbb174be378c0b45f30836e74aaaf3919b5e-filelists.sqlite.bz2 from updates: [Errno 256] No more mirrors to try. You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest

Fixed by googling and running this:
yum clean metadata yum clean dbcache yum update

Win7 updates won’t work

Leave a reply

Fresh install of Win7 (refurbished) and updates won’t work, no matter which MS FixIt tools you try? Put this into a .bat file and run it as Administrator:

net stop wuauserv net stop CryptSvc ren %windir%\system32\catroot2 catroot2.old ren %windir%\SoftwareDistribution sold.old net start CryptSvc net start wuauserv pause