Debugging IDM

One of the most frequent cases I have is that “sometimes” and “somewhere” user is not getting authenticated. Trying to SSH to the host works for some users not always, “id username” returns errors sometimes — it’s all the same problem in the environment with LDAP replication. It does not actually matter what kind of the LDAP server or domain controller is being used, always check:

    1. Enable debug log on the client. If the client is SSSD, add “debug_level = 9” to the /etc/sssd/sssd.conf and then restart it. Invalidate its cache if possible.
    2. Repeat the test so you would see the error.
    3. Collect the log file from the client. You would see what server it has queried to get the information.
    4. Check server’s log. Most likely there is no requested information on this LDAP instance due to replication issues.

This would help to identify and fix the problem.

Parsing sssd debug log

Lol, hope to add more in furure

grep -v "timed event" |grep -v "timer event"|grep -v "Requesting"|grep -v "SBUS"|grep -v "callback"|grep -v "dispatch"|grep -v "a sysbus message"|grep -v "No sub-attributes for" |grep -v "reusing cached connection" |grep -v "nesting:"|grep -v "sbus_remove_watch"|grep -v "be_client_destructor"|grep -v "sdap_process_result"|grep -v "Comparing LDAP with LDAP" |grep -v "Message type:"|grep -v "unenforced gpo skipped"

More .bashrc improvements

I need to see long path and host name. And not to lose command line space at the same time:

BOLD="\[$($TTY  && /usr/bin/tput bold)\]"
COLOR_BLACK="\[$($TTY  && /usr/bin/tput setaf 0)\]"
COLOR_RED="\[$($TTY  && /usr/bin/tput setaf 1)\]"
COLOR_GREEN="\[$($TTY  && /usr/bin/tput setaf 2)\]"
COLOR_YELLOW="\[$($TTY && /usr/bin/tput setaf 3)\]"
COLOR_BLUE="\[$($TTY  && /usr/bin/tput setaf 4)\]"
COLOR_MAGENTA="\[$($TTY  && /usr/bin/tput setaf 5)\]"
COLOR_CYAN="\[$($TTY  && /usr/bin/tput setaf 6)\]"
COLOR_WHITE="\[$($TTY  && /usr/bin/tput setaf 7)\]"
COLOR_GRAY="\[$($TTY  && /usr/bin/tput setaf 8)\]"
RESET="\[$($TTY  && /usr/bin/tput sgr0)\]"



function prompt_command  {     
    local TIMESTAMP="${BOLD}${COLOR_GRAY}\D{%d/%m %H:%M:%S}${RESET}"
    local CURPWD="${BOLD}${COLOR_BLUE}${PWD/#${HOME}/~}${RESET}"

    PS1="\n┌[${BOLD}\u@${COLOR_RED}\h${RESET}]─[${CURPWD}]\n"
    PS1=${PS1}"└─"${TIMESTAMP}${BOLD}"-> "

}

PROMPT_COMMAND=prompt_command

Google group calendars in Evolution Mail

If your organization uses Google Apps as mail service but you are using Evolution, there is no evident way to view and edit shared group calendars until recent versions. To add group calendar you need:

  1. go to calendar settings on web
  2. Calendar Address: -> ID (somenting like example.com_1d32345331343234213832@resource.calendar.google.com)
  3. Evolution -> New Calendar -> Google
  4. User name -> this ID
  5. Auth with usual name-pass or other means you use (Kerberos, OTP) in the window appear.
  6. Allow Evolution to use the Google’s resources.

DKIM=temperror

Recently I have checked my messages headers of the mail that comes from the andreybondarenko.com MX and found that Google shows that my DKIM signature is invalid:

Authentication-Results: mx.google.com; dkim=temperror (no key for
 signature) header.i=@andreybondarenko.com;

In the spam score section:

DKIM_SIGNED,T_DKIM_INVALID

However the header itself is present:

DKIM-Filter: OpenDKIM Filter v2.10.3 andreybondrenko.com CE25780BAC
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=andreybondarenko.com; s=default; t=1483634085;
        bh=w00tuUhwty0/5n/YHiopiY3PpnqKT5BLK9l6TkDNUUk=;
        h=Subject:From:Reply-To:To:Date:From;
        b=J5qB5RF9lrOho1wBpLyLi5a6CwIHZK1sugCr2wpwnPKwEg76RFv2/y8xaiwquqftX
         VhTJH9NLJXcPdu8k8/zN/sc8P1RksNR9EvDw6k2YNEKoeMsKMGgyMC4kAAhcT31IgX
         eqnIqWxhVTVdjRqrqzNPn0wuBbGJgO2bwmFcVsy8=

I have found that it’s quite common configuration error of the OpenDKIM, the selector you choose to store key can be chosen randomly, but the TXT record should match /etc/opendkim.conf. In my case:

##  Defines the name of the selector to be used when signing messages.
Selector   default

But the DNS record:

[user@andreybondarenko ~]$ dig TXT mail._domainkey.andreybondarenko.com

;; ANSWER SECTION:
mail._domainkey.andreybondarenko.com. 1800 IN TXT "v=DKIM1\; k=rsa\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCdGRWtWPPZVIg0fy7Pr0+rsBsoL6Imt1GBE/QRd3X5Izv1iAJFUsOtea
f9TI9EO/YFwoLLahzuoZM1oUU4ED3fHlItEnqXCKQhX8Zripi7gfIO+DRFEhGuQtG6OIuA6+c3ivao7DTPk/IFqY7MG5M3wMvAfV+
eIBf1VjmajSwe3wIDAQAB"

Changing ‘Selector’ to ‘mail’ and restating opendkim (it’s faster then change DNS):

Authentication-Results: mx.google.com; dkim=pass
 header.i=@andreybondarenko.com;

In the spam score section:

DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU

Let’s encrypt cert updates

Let’s encrypt is wonderful, but certificate are getting expired every 3 months. Since it’s a first time I need to renew them, I have done it manually. The tool authenticates you (by default) with special file created in the .well-know/acme-challenge directory of the root, so the blog engine should not interfere or rewrite anything and should not return it’s own 404 page. Historically my nginx.conf has lots of existing redirects and rules, I am too lazy to correct and simplify it, so simple

localtion ~ .well-known {
        allow all;
    }

does not work. And I am too lazy to figure out why it is so (bad for me). So the most simple way to renew certs for me is to switch to minimal config. Putting it here for the future reference.

user  nginx;  
worker_processes  1;
error_log  /var/log/nginx/error.log;
pid        /run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    server_names_hash_bucket_size 128;
    index   index.html index.htm;
    server { listen 80;
        listen [::]:80;
        server_name andreybondarenko.com;
location / {
                 root   /var/www/;
        }

    }
}

How to sort messages log

If you need to sort out what is reporting to the /var/log/messages to array in case of Red Hat Enterprise Linux or Fedora, you need to do flowing manipulations:

  1. read log;
  2. get 5th column from the log, it’s daemon name;
  3. get rid of all digits, so the daemons with different PIDs would be counted as one;

1. get rid of all ‘/’ and replace ‘[’ and ‘]’ with ‘\[’ and ‘\]’ to keep things both readable and usable for future scripting;

  1. sort unique stuff.

in my case this gets look like:

cat /var/log/messages | awk '{ print $5 }'| sed 's/\[[0-9].*$//'|sed 's/\[/\\\[/g' | sed 's/\]/\\\]/g'| sed 's/://g' |sed 's/\///g' |sort -u``

the result is usable as array for example. My host’s result:

abrt-hook-ccpp
at-spi-bus-launcher
audit
avahi-daemon
blueman.desktop
blueman-mechanism
bluetoothd
chronyd
cinnamon-killer-daemon
cinnamon-session
com.redhat.imsettings
...
skip
...
tracker-store.desktop
udisksd
usrlibexecgdm-x-session
vmware-user.desktop
wpa_supplicant

Debugging Kerberos

If you need to debug Kerberos, check the time synchronization at the first place. In about 50% cases it is it.

  1. the ntpd (or chrony) should be presented in the process list
  2. they should really be configured correctly
  3. in case of the virtual host crony is preferable, with the ntpd time skew is possible

Really nice crony/ntpd comparative chart: https://chrony.tuxfamily.org/comparison.html , “Summary” section is complete.