How to change the exported LUN number on a VIOS «Virtual Target Disk» or VTD

When you create a «Virtual Target Disk» or VTD on a VIOS, there is not documented way to define or change the LUN number that it shows to the client partition. But there are situation where you might need to update it:

1. In a dual VIOS environment, to have the same LUNs in both clients (easier to administrate)-
2. In a redudant configuration, when you need to start lpars on different hardware, using SAN disks. For instance, we use this configuration for our Backup datacenter where we have al the SAN disks mirrored.

In this post I comment how to update this LUN. The idea is basicly:

• Set the VTD device to Defined in the VIOS
• Update the ODM database. You have to update the attribute ‘LogicalUnitAddr’ in ObjectClass ‘CuAt’
• Perform a ‘cfgmgr’ on the virtual host adapter (vhostX). This will enable the VTD device and reload the LUN number. Perform an cfgmgr on the VTD device does not work.

So, with commands:

$ oem_setup_env
# bash

# lsmap -vadapter vhost21
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost21         U9117.MMA.XXXXXXX-V2-C34                     0x00000016

VTD                   host01v01
Status                Available
LUN                   0x8200000000000000
Backing device        hdiskpower0
Physloc               U789D.001.BBBBBBB-P1-C3-T2-L75

# ioscli mkvdev -vadapter vhost21 -dev host01v99 -vdev hdiskpower1
cfgmgr -l vhost21

# lsmap -vadapter vhost21
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost21         U9117.MMA.XXXXXXX-V2-C34                     0x00000016

VTD                   host01v01
Status                Available
LUN                   0x8200000000000000
Backing device        hdiskpower0
Physloc               U789D.001.JJJJJJJ-P1-C3-T2-L75

VTD                   host01v99
Status                Available
LUN                   0x8300000000000000
Backing device        hdiskpower1
Physloc               U789D.001.JJJJJJJ-P1-C3-T2-L77

# rmdev -l host01v99
host01v99 Defined

# odmget -q "name=host01v99 and attribute=LogicalUnitAddr"  CuAt
CuAt:
  name = "host01v99"
  attribute = "LogicalUnitAddr"
  value = "0x8300000000000000"
  type = "R"
  generic = "D"
  rep = "n"
  nls_index = 6

# odmchange -o CuAt -q "name = host01v99 and attribute = LogicalUnitAddr" <<"EOF"
CuAt:
  name = "host01v99"
  attribute = "LogicalUnitAddr"
  value = "0x8100000000000000"
  type = "R"
  generic = "D"
  rep = "n"
  nls_index = 6
EOF

# odmget -q "name=host01v99 and attribute=LogicalUnitAddr"  CuAt
CuAt:
  name = "host01v99"
  attribute = "LogicalUnitAddr"
  value = "0x8100000000000000"
  type = "R"
  generic = "D"
  rep = "n"
  nls_index = 6

# cfgmgr -l vhost21
# lsmap -vadapter vhost21
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost21         U9117.MMA.XXXXXXX-V2-C34                     0x00000016

VTD                   host01v01
Status                Available
LUN                   0x8200000000000000
Backing device        hdiskpower0
Physloc               U789D.001.JJJJJJJ-P1-C3-T2-L75

VTD                   host01v99
Status                Available
LUN                   0x8100000000000000
Backing device        hdiskpower1
Physloc               U789D.001.JJJJJJJ-P1-C3-T2-L77

In the client partition, you can scan for the new disk, and it will have the LUN 0x81:

root@host01:~/# cfgmgr -l vio0
root@host01:~/# lscfg -vl hdisk5
  hdisk5           U9117.MMA.XXXXXXX-V22-C3-T1-L8100000000000000  Virtual SCSI Disk Drive

Note: Actually I changed the output of these commands to remove information of my company.

Update: I created an script to do this: change_vtd_lun.sh

AIX vs GNU core utils tools: the ‘dd’ command with skip parameter.

Any Linux & Unix admin knowns this fact: GNU tools are MUCH MORE better tools than AIX, BSD, Solaris or HP-UX tools.

GNU tools have much less bugs, much more functionality and options, localization, better documentation, they are standard, most of the scripts are built based on GNU tools, etc, etc,etc. Why the hell they do not throw out their ugly-buggy-limitated tools and install the GNU tools in their systems by default???

Here you have an example of a weird behaviour in the ‘dd’ command in the AIX platform: With the skip=<Num. blocks> parameter the ‘dd’ command skips the blocks, but it actually reads them (no matter if you are working on a filesystem with file random access). So, if you are working with big files (in my case, 50GB) you have to read ALL the blocks in memory before access the requested position. That means huge I/O, usage of memory in cache, etc…

IBM guys: you do not know that there is a lseek(2) function?

Here you have an example of the time that takes read 2MB from a big file, skiping 1000MB. Using native ‘dd’ command takes 12s:

$ time /usr/bin/dd if=a_big_big_file.data skip=1000 bs=1M count=2 of=/dev/null
2+0 records in.
2+0 records out.

real    0m12.059s
user    0m0.013s
sys     0m1.419s

With GNU’s version, less than a second:

$ time /opt/freeware/bin/dd if=a_big_big_file.data skip=1000 bs=1M count=2 of=/dev/null
2+0 records in
2+0 records out

real    0m0.024s
user    0m0.002s
sys     0m0.006s

Note: You can find the GNU’s dd tool in AIX Linux ToolBox coreutils package.

Update: I contacted the IBM support and they told me that using the option conv=iblock ,»dd» will behave as expected. But IMHO the documentation does not explicitily say that:

iblock, oblock

Minimize data loss resulting from a read or write error on direct access devices. If you specify the iblock variable and an error occurs during a block read (where the block size is 512 or the size specified by theibs=InputBlockSize variable), the dd command attempts to reread the data block in smaller size units. If the dd command can determine the sector size of the input device, it reads the damaged block one sector at a time. Otherwise, it reads it 512 bytes at a time. The input block size (ibs) must be a multiple of this retry size. This option contains data loss associated with a read error to a single sector. The oblock conversion works similarly on output.

 

Los envíos de IBM y su «ecologismo»

Desde que trabajo con IBM siempre me sorprendieron los envios de documentación y actualizaciones de IBM. Ya no es que rocen el absurdo, sino que lo sobrepasan y lo redefinen en un nuevo concepto de ridiculez.

Por ejemplo, ahora, por iniciativa propia del soporte de IBM, han decidido mandarme los últimos parches de AIX 6.1 TL5. Se pusieron en contacto conmigo y aunque les dije que no hacía falta y que me lo bajaba de internet, me lo mandaron igual… y esto es lo que me mandaron:

• AIX TL5, Corrective Service (ni siquiera los base)
  • 47 CDs
  • listado de 120 hojas con todos los ficheros y parches que incluye
• AIX TL5 SP1, primer nivel de SP
  • 42 CDs
  • listado de 130 hojas con todos los ficheros y parches que incluye
• AIX TL5 SP2, segundo nivel de SP
  • 44 CDs
  • listado de 130 hojas con todos los ficheros y parches que incluye

Casi nada… y además PARA NADA, porque yo ni de coña voy a usar eso si tengo un mecanismo de descarga y aplicación de parches.

Aún recuerdo cuando me mandaron una caja de 30x20x10, con varios niveles de cajas y cartones para incluir una única hoja con recomendaciones de seguridad física: cómo usar una pulsera estática o que no se debe mirar a la lente del CD-ROM :-D

¿y luego esta gente presume de ecologista?

Configure a cups-pdf printer on Cups and use it on AIX

I will briefly describe how to set a cups-pdf on cups on Linux and configure AIX to use it. It is an easy task.

1. Install on Linux cups and cups-pdf (for SucksE (Suse) you can find it in openSuse repositories).

   The cups-pdf package configures automaticly a printer called «cups-pdf»

2. You can access the CUPS configuration page via http://localhost:631. If it is in a remote server, you can forward the port via SSH: «ssh -R 6310:localhost:631 host» and access via http://localhost:631.
3. To use it on AIX, you need to configure the LPD protocol enabling cups-lpd in xinetd: On suse you must enable it in /etc/xinetd.d/cups-lpd.NOTE: You must disable the usage of banners (added by default by cups-lpd when converting from lpd to ipp) or you will get always a file called «Test_Page.pdf» with only the banner. I think that newer versions of cups solve this problem. To do that, you must add to cups-lpd the option -o job-sheets=none

sed 's/\(disable.*=\).*/\1 no/' -i /etc/xinetd.d/cups-lpd	
grep -q job-sheets=none /etc/xinetd.d/cups-lpd || sed 's/\(server_args.*=.*\)/\1 -o job-sheets=none/' -i /etc/xinetd.d/cups-lpd
/etc/init.d/xinetd reload

Finally on AIX, you can create you new printer as a BSD printer:

/usr/lib/lpd/pio/etc/piomisc_ext mkpq_remote_ext  -q 'cups-pdf' -h 'remoteserver' -r 'cups-pdf' -t 'bsd' -C 'FALSE' -d 'Virtual PDF printer on remoteserver'

That is all. You can use your virtual pdf printer on AIX: ls | lp -d cups-pdf

You may want tune some cups-pdf settings in /etc/cups/cups-pdf.conf, like:

• UserUMask 0007: This option affects the «umask» default ACL configuration. If you set 0077 it will set umask=— in final PDF, I do not known why :?

  ### Key: UserUMask
  ##  umask for user output of known users
  ##  changing this can introduce security leaks if confidential
  ##  information is processed!
  ### Default: 0077
  
  UserUMask 0007
  

• Label 1, to avoid overwrites…

  ### Key: Label
  ##  label all jobs with a unique job-id in order to avoid overwriting old
  ##  files in case new ones with identical names are created; always true for
  ##  untitled documents
  ##  0: label untitled documents only, 1: label all documents
  ### Default: 0
  

• Paths, etc….

Extending puppet for AIX: manage nfs mountpoints

I am playing arourd with puppet, a configuration management software.

We have both AIX and Linux systems, but I find puppet a little bit inmature for AIX.

Anyway, I think that it will be easy implement providers and recipes using the Exec providers on AIX. AIX has a consistent set of commands, and almost everything can be configured from command line. Most of commands have similar options, syntax and ouput.

Normally, all OS configuration items (users, groups, mountpoints) have a set of commands: ls, ch and mk. Most of then are based on attributes that can be changed/set and output of ls* commands usually can be printed in colons (-c option).

How does AIX manage users defined in different DB? Some Gotchas.

We want to known who will work AIX with duplicated users and groups in the BUILTIN and LDAP databases.

In Linux, with NSS, the OS follows the rules defined in /etc/nsswitch.conf, and merges the credentials. If two users entries same name and different id or vice versa, it will get the first one. But in AIX is different.

AIX: Using locally defined users with Kerberos auth (KRB5files).

You want to have a user defined locally but delegate the authentication to a Kerberos server (like active directory). That is ok, specially since secldapclntd is not the most reliable component on aix.

But be careful, if you define a user in the compat registry instead of KRB5files (but with SYSTEM=KRB5files), like in this command:

mkuser -R KRB5files SYSTEM=KRB5files <user>

you will find that the local password policies will be applied to the user. This is a incorrect behaviour, because AIX does not manage the password.

For instance, despite having SYSTEM=KRB5files, the new user will have the ADMCHG attribute defined in its stanza in /etc/security/passwd

jhon:
        password = *
        lastupdate = 1275046476
        flags = ADMCHG

From man pwdadm:

ADMCHG
   Resets the ADMCHG attribute without changing the user's password. This forces the user to change passwords
   the next time a login command or an su command is given for the user. The attribute is cleared when the
   user specified by the User parameter resets the password.

With this attribute set and SYSTEM=KRB5files, we will get this error if we try to login (for instance, via SSH):

May 31 10:10:38 aixhost01 auth|security:info sshd[585730]: Password can't be changed for user jhon: [compat]: 3004-333 A password change is required. 3004-320 Only the system administrator can change
May 31 10:10:38 aixhost01 auth|security:info sshd[585730]: Failed password for jhon from 1.2.3.4 port 62018 ssh2
May 31 10:10:38 aixhost01 auth|security:info syslog: ssh: failed login attempt for jhon from acomputer.localdomain

To avoid this, you can reset the password, or execute pwdadm -c jhon, but the best solution is simply change the registry:

chuser registry=KRB5files jhon

Or:

USER=jhon
chuser expires=0 maxage=0 maxexpired=-1 minage=0 loginretries=-1 registry=KRB5files $USER
pwdadm -c $USER

Other tipical errors of this:

Sep 15 09:56:56 tcmurexappl1 auth|security:info sshd[344552]: Password can’t be changed for user jhon: [compat]: 3004-330 Yourencrypted password is invalid. \r3004-320 Only the system administrator can c
Sep 15 09:56:14 tcmurexappl1 auth|security:info sshd[213376]: Failed password for jhon from 168.1.1.x port 1632 ssh2
Sep 15 09:56:14 tcmurexappl1 auth|security:info syslog: ssh: failed login attempt for jhon from apc.mycompay.org
Sep 15 09:56:18 tcmurexappl1 auth|security:debug sshd[213376]: [krb_authenticate] Error in getting TGT …
Sep 15 09:56:18 tcmurexappl1 auth|security:debug sshd[213376]: Preauthentication failed

To solve it:

USER=jhon
chuser expires=0 maxage=0 maxexpired=-1 minage=0 loginretries=-1 registry=KRB5files $USER
pwdadm -c $USER

Script to clonate ldap users to local users in AIX

AIX LDAP integration is not up to expectations. Its cache daemon, secldapclntd,
has a lot of problems:it often crashes, queries are slow, etc…

To mitigate problems, one workaround could be create the most important users locally,
using the KRB5files repository.

With this idea, this script will query a set of given groups from the AIX LDAP
registry using the AIX command line tools (lsuser, lsgroup), and it will create
them locally (mkgroup, mkuser).

To make it work, the host must be integrated with remote repository and must be able to resolve users and groups with LDAP method. You need LDAP method and KRB5files method configured. It can be easily changed to use other methods.

This script also supports nested groups from Active Directory.

• Source: http://github.com/keymon/snippets/blob/master/scripts/aix/clone-aix-ldap-groups.sh
• Download: http://github.com/keymon/snippets/raw/master/scripts/aix/clone-aix-ldap-groups.sh

Installing OpenSSH and OpenSSL on AIX 6.1

There are a lot of sites where the process is explained, just google a little bit.

I will describe the one that worked for me:

• I am using AIX 6.1 TL4 (upgraded from AIX 5.3 TL6)
• I have the AIX Toolbox For Linux Applications of 05.2009

I prefer to have openssh and openssl as native AIX packages. The problem is that .rpm files usually have dependencies with openssl, so I had to install the openssl rpm package as well.

First I downloaded both openssl 0.9.8.1103 (from IBM) and openssh 5.2p1 (from Sourceforge):

• https://www14.software.ibm.com/webapp/iwm/web/reg/download.do?source=aixbp&S_PKG=openssl
• http://sourceforge.net/projects/openssh-aix/files/openssh-aix61/openssh_5.2p1_aix61.tar.Z/download

I installed it:

mkdir openssh_5.2p1_aix61 && cd openssh_5.2p1_aix61 && uncompress -c < ../openssh_5.2p1_aix61.tar.Z |tar -xvf - && installp -acXYgd . openssh
mkdir openssl.0.9.8.1103 && cd openssl.0.9.8.1103 && uncompress -c < ../openssl.0.9.8.1103.tar.Z |tar -xvf - && installp -acXYgd . openssl


Then I downloaded a compiled version of openssl 0.9.8 from perzl.org (AIX Toolbox For Linux Applications of 05.2009 comes with 0.9.7, but .rpms have dependencies on 0.9.8): http://www.perzl.org/aix/index.php?n=Main.Openssl. And I installed it rpm -i openssl-0.9.8n-1.aix5.1.ppc.rpm.

Monitoring and restarting secldapclntd daemon in AIX

The AIX service secldapclntd «Provides and manages connection between the AIX LDAP load module of the local host and LDAP Security Information Server, and handles transactions from the LDAP load module to the LDAP Security Information Server.»

This services fails too often. Each new version of AIX, brings new failures in this services. Failures appear more often if the LDAP server has a lot of users and groups.

When it fails:

• sometimes does not reply, its hung
• sometimes it consumes all CPU and lasts a lot to reply
• sometimes it simply dies with a core.

This script will check and monitor it and restart it if necesary.

You can test this script stoping the service:

kill -STOP $(ps -fea| grep -v grep |grep /usr/sbin/secldapclntd| awk '{print $2}' )

You can add an entry to cron to execute it:

if crontab -l | grep /usr/local/bin/check-secldapclntd.sh; then
   echo "Already configured."
else
  crontab -l > /tmp/$$.crontab
  cat >> /tmp/$$.crontab <<EOF
# Check secldapclntd each 5 minutes
5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/local/bin/check-secldapclntd.sh check-and-restart > /dev/null
EOF
  crontab /tmp/$$.crontab
  rm /tmp/$$.crontab
fi

And here goes the script (/usr/local/bin/check-secldapclntd.sh):

• Source: http://github.com/keymon/snippets/blob/master/scripts/aix/check-secldapclntd.sh
• Download (Right Click > save as…): http://github.com/keymon/snippets/raw/master/scripts/aix/check-secldapclntd.sh