Archivos de la categoría ‘script’

Hace un porrón de años escribí esto… creo que aún estaba en el instituto :)

Básicamente funciona así (se ve mal por culpa de wordpress que quita los espacios… paso de rayarme):

$ fortune | ./papel.sh
 __,-----------------------------------------------------------------------,__
/  \\                                                                       \ \
\__/|       Many a writer seems to think he is never profound except         |/
 __ |       when he can't understand his own meaning.                        |
/  \|       -- George D. Prentice                                            |\
\__//                                                                        //
   `------------------------------------------------------------------------``
#!/bin/sh
# Poner en el cron:
# 33 *     * * *     root   (cat /etc/motd_base ; /usr/games/fortune | /usr/local/bin/papel.sh) > /etc/motd; chmod 644 /etc/motd

fmt -w 60 | (
read a;
read b;
read c;

cat << "TOP"
 __,-----------------------------------------------------------------------,__
/  \\                                                                       \ \
TOP

printf "\\__/|       %-65s|/\n" "$a"
a=$b;
b=$c;
read c;

while read c; do
	printf "    |       %-65s|\n" "$a";
	a=$b;
	b=$c;
done;

printf " __ |       %-65s|\n" "$a";
printf "/  \\|       %-65s|\\ \n" "$b";

cat << "BOTTON"
\__//                                                                        //
   `------------------------------------------------------------------------``
BOTTON

)

These is one of the proposed solutions for the job assessment commented in a previous post.

Provide a design which is able to parse Apache access-logs so we can generate an overview of which IP has visited a specific link. This design needs to be usable for a 500+ node webcluster. Please provide your configs/possible scripts and explain your choices and how scalable they are.

I will consider these requirements:

  • It is not critical to register all the log entries. It is no needed ensure that all the web hits are registered.
  • No control on duplicated log entries. It is not needed
    to check that the log entries had been already loaded.
  • It is also needed to propose a mechanism to gather the logs from the webservers.
  • It must be scalable.
  • It is a plus to make it flexible to allow further different analysis.

The problems to be solved are log storage and log gathering, but the main bottleneck will be the storage.

One realizes that the best option is a noSQL database due to
the characteristics of the data to process (log entries):

  • Time ordered entries
  • no duplicates
  • need of fast insertion
  • fixed fields
  • no data relation or conceptual integrity
  • need to be rotated (old entries removed)
  • etc…

So, I will propose the usage of MongoDB [1] (http://www.mongodb.org/), that fits the requirements:

  • It is fast, both at inserting and querying.
  • Scales horizontally without disruption (is initially proper configured).
  • Supports replication and High Availability.
  • Well known solution. Commercial support if needed.
  • Python bindings (pyMongo)
[1] Note: I will not enter in details of a MongoDB scalable HA architecture.
See the quick start guide
to setup a single node and the documentation
for architecture examples.

To parse the logs and store them in MongoDB, I will propose a
simple python script: accesslog_parse_mongo.py that:

  • Setup a direct MongoDB connection.
  • Read the access log from standard input.
  • Parse the logs and store all the fields, including: client_ip, url, referer, status code, timestamp, timezone…
  • I do not set any indexes in the NoSQL db. Indexes could be
    created on url or client_ip fields, but not having indexes allows faster
    insertions, that is the objective. The reads are very uncommon and performed
    in batch processes.
  • Notice that it should be improved to be more reliable. For instance, it
    does not check for errors (DB failures, etc.). It could buffer entries in case of DB failure.
  • A second script called example_query_accesslog.py queries the DB and prints the access. It gets an optional argument, the relative URL.

To feed the DB with the logs from the webservers, some solutions could be:

  • Copy the log files with a scheduled task via SSH or similar, then process them with
    accesslog_parse_mongo.py in a centralized server (or cluster of servers).

    • Pros: Logs are centralized. Only a set of servers access to MongoDB.
      System can be stopped as needed.
    • Cons: Needs extra programming to get the logs.
      No realtime data.
  • Use a centralized syslog service, like syslog-ng
    (can be balanced and configured in HA),
    and setup all the webservers to send the logs via syslog
    (see this article).

    In the log server, we can process resulting files with a batch process or send all the messages to accesslog_parse_mongo.py. For instance, the configuration for syslog-ng:

    destination d_prog { program("/apath/accesslog_parse_mongo.py"
                                  template(“$MSGONLY\n”)
                                  template-escape(no)); };
    • Pros: Centralized logs. No extra programming. Realtime data.
      Use of existent infrastructure (syslog). Only a set of servers access to MongoDB.
    • Cons: Some logs entries can be dropped. Can not be stopped, if not log entries will be lost.
  • Pipe the webserver logs directly to the script, accesslog_parse_mongo.py.
    In Apache configuration:

    CustomLog "|/apath/accesslog_parse_mongo.py" combined
    • Pros: Easy to implement. No extra programming or infrastructure. Realtime data.
    • Cons: Some logs entries can be dropped. It can not be stopped or log entries will be lost.
      The script should be improved to make it more reliable.

These is one of the proposed solutions for the job assessment commented in a previous post.

Based to this post, http://linux-tips.org/article/78/syntax-highlighting-in-less my fast tip to allow syntax highlight in less:

cat <<EOF >> ~/.bash_profile

# Syntax Highlight for less
#
# Check if source highlite is intalled http://www.gnu.org/software/src-highlite/
# Set SRC_HILITE_LESSPIPE for custom location
# 
# To install: 
#   sudo yum install source-highlight
#
SRC_HILITE_LESSPIPE=${SRC_HILITE_LESSPIPE:-$(which src-hilite-lesspipe.sh 2> /dev/null)}
if [ -x "$SRC_HILITE_LESSPIPE" ]; then
	export LESSOPEN="| $SRC_HILITE_LESSPIPE  %s"
	export LESS="${LESS/ -R/}  -R" # Set LESS option in raw mode (avoid repeating it)
fi
EOF

Sometimes you need to know who was the user that did login in a linux/unix server, but after several “sudo” or “su” commands (and others programs that change the permissions) you have lost the information.

You can try to determine the user using the tty of the process tree, querying the process parents.

With this idea, I wrote this small script: whowasi.sh

#!/bin/env bash
# This scripts allows  determine the user used to login in the
# machine to run the given process.
# 
SCRIPT_NAME=$0

# Command names to be considered as login commands
LOGIN_PROGRAMS="sshd telnetd login" 

# Get all pids of the parents of a pid
get_parent_pids() {
    echo $1
    ppid=$(ps -o ppid -p $1 | awk '/[0-9]+/ {print $1}' )
    [ $ppid == 1 -o $ppid == 0 ] && return
    get_parent_pids $ppid
}

# Get users of parent process of a pid
get_parent_users() {
	get_parent_pids $1 | xargs -n1 ps -o user= -p | uniq | awk '{print $1}'
}

get_parent_users_commands() {
	get_parent_pids $1 | xargs -n1 ps -o user= -o comm= -p | uniq
}

get_parent_users_ttys() {
	get_parent_pids $1 | xargs -n1 ps -o user= -o tty= -p | uniq
}


get_firstuser_after_login() {
	cmd="egrep -B1 -m1" # Get the line before, and stop on first match
	for p in $LOGIN_PROGRAMS; do 
		cmd="$cmd -e '^(.*/)?$p\$'" 
	done
	get_parent_users_commands $1 | eval $cmd | awk '{ print $1; exit; }'
}

get_firstuser_after_root() {
	get_parent_users $1 | grep -B1 -m1 root | awk '{print $1;exit;}'
}

get_firstuser_with_tty() {
	get_parent_users_ttys $1 | grep -B1 -m1 \?  | awk '{print $1;exit;}'
}


print_help() {
	cat <<EOF
Usage $SCRIPT_NAME [Option...] [pid]

Prints the users that where used to start a process.

By default it will use the current process.
	
Options
	-h:		This help.
	-t:		Print the user of the first process having a valid tty (not ?) 
			This is the default behaviour.
	-a:		Print all processes.
	-r:		Print only user started after the first root (usually the one that login in)
	-l:		Print the user after a login program ($LOGIN_PROGRAMS)
	        Requires GNU egrep.
EOF
}

mode=tty
while true; do
	case $1 in
		"")
			break
		;;
		"-a")
			mode=all
		;;
		"-l")
			mode=login
		;; 
		"-r")
			mode=root
		;;
		"-t")
			mode=tty
		;;
		"-h")
			printhelp
			exit
		;;
		"-*")
			echo "$SCRIPT_NAME: Unknown option '$1'"
			printhelp
			exit
		;;
		*)
			args="$args $1"
		;;
	esac
	shift
done
set -- $args

pid=${1:-$$}

if ! ps -p $pid >/dev/null 2>&1; then
	echo "$SCRIPT_NAME: Unable to find process '$pid'"
	exit 1
fi

case $mode in 
	all)
		get_parent_users $pid
	;;
	login)
		get_firstuser_after_login $pid
	;;
	root)
		get_firstuser_after_root $pid
	;;
	tty)
		get_firstuser_with_tty $pid
	;;
esac
When you create a “Virtual Target Disk” or VTD on a VIOS, there is not documented way to define or change the LUN number that it shows to the client partition. But there are situation where you might need to update it:
  1. In a dual VIOS environment, to have the same LUNs in both clients (easier to administrate)-
  2. In a redudant configuration, when you need to start lpars on different hardware, using SAN disks. For instance, we use this configuration for our Backup datacenter where we have al the SAN disks mirrored.

In this post I comment how to update this LUN. The idea is basicly:

  • Set the VTD device to Defined in the VIOS
  • Update the ODM database. You have to update the attribute ‘LogicalUnitAddr’ in ObjectClass ‘CuAt’
  • Perform a ‘cfgmgr’ on the virtual host adapter (vhostX). This will enable the VTD device and reload the LUN number. Perform an cfgmgr on the VTD device does not work.

So, with commands:

$ oem_setup_env
# bash

# lsmap -vadapter vhost21
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost21         U9117.MMA.XXXXXXX-V2-C34                     0x00000016

VTD                   host01v01
Status                Available
LUN                   0x8200000000000000
Backing device        hdiskpower0
Physloc               U789D.001.BBBBBBB-P1-C3-T2-L75

# ioscli mkvdev -vadapter vhost21 -dev host01v99 -vdev hdiskpower1
cfgmgr -l vhost21

# lsmap -vadapter vhost21
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost21         U9117.MMA.XXXXXXX-V2-C34                     0x00000016

VTD                   host01v01
Status                Available
LUN                   0x8200000000000000
Backing device        hdiskpower0
Physloc               U789D.001.JJJJJJJ-P1-C3-T2-L75

VTD                   host01v99
Status                Available
LUN                   0x8300000000000000
Backing device        hdiskpower1
Physloc               U789D.001.JJJJJJJ-P1-C3-T2-L77

# rmdev -l host01v99
host01v99 Defined

# odmget -q "name=host01v99 and attribute=LogicalUnitAddr"  CuAt
CuAt:
  name = "host01v99"
  attribute = "LogicalUnitAddr"
  value = "0x8300000000000000"
  type = "R"
  generic = "D"
  rep = "n"
  nls_index = 6

# odmchange -o CuAt -q "name = host01v99 and attribute = LogicalUnitAddr" <<"EOF"
CuAt:
  name = "host01v99"
  attribute = "LogicalUnitAddr"
  value = "0x8100000000000000"
  type = "R"
  generic = "D"
  rep = "n"
  nls_index = 6
EOF

# odmget -q "name=host01v99 and attribute=LogicalUnitAddr"  CuAt
CuAt:
  name = "host01v99"
  attribute = "LogicalUnitAddr"
  value = "0x8100000000000000"
  type = "R"
  generic = "D"
  rep = "n"
  nls_index = 6

# cfgmgr -l vhost21
# lsmap -vadapter vhost21
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost21         U9117.MMA.XXXXXXX-V2-C34                     0x00000016

VTD                   host01v01
Status                Available
LUN                   0x8200000000000000
Backing device        hdiskpower0
Physloc               U789D.001.JJJJJJJ-P1-C3-T2-L75

VTD                   host01v99
Status                Available
LUN                   0x8100000000000000
Backing device        hdiskpower1
Physloc               U789D.001.JJJJJJJ-P1-C3-T2-L77

In the client partition, you can scan for the new disk, and it will have the LUN 0x81:

root@host01:~/# cfgmgr -l vio0
root@host01:~/# lscfg -vl hdisk5
  hdisk5           U9117.MMA.XXXXXXX-V22-C3-T1-L8100000000000000  Virtual SCSI Disk Drive

Note: Actually I changed the output of these commands to remove information of my company.

Update: I created an script to do this: change_vtd_lun.sh

Puppet architecture needs a client to connect to the server to load the configuration usin a pull schema. But I do not like to have more and more daemons around and some people suggest avoid that , so I decided to execute puppet using ‘–onetime’ option from cron.

Obviously, I want to configure this using puppet itself. And we must ensure that the clients are executed at different times, not all at the same minute.

I searched the net and I found several aproaches to do this. There are also feature requests.

I read somewhere that the new function fqdn_rand() could be used, as proposed in the feature request and posted in this mail from Brice Figureau. I can not find where the hell the snippet was. At the end, I found this pastie by Jhon Goebel.

I will post my version here just to keep it wrote down.

 $first = fqdn_rand(30)
 $second = fqdn_rand(30) + 30
 cron { "cron.puppet.onetime":
 command => "/srv/scripts/puppet/puppet.ctl.sh onetime > /dev/null",
 user => "root",
 minute => [ $first, $second ],
 require => File["/srv/scripts/puppet/puppet.ctl.sh"],
 }

… this is another random thinking from keymon (https://keymon.wordpress.com)


									

Quiero presentar github (http://github.com/), un site gratuito para hosting de código fuente (publico o privado) en internet. Con todas las ventajas de git, pero “en la nube” y gratis :).

Lo he estado probando y parece que cubre perfectamente mis necesidades:

  • Puedo tener un repositorio online sin límites.
  • Permite consultar y enlazar los fuentes, con iluminación de síntaxis o en modo ‘raw’
  • Puede ser accesible desde detrás de un proxy que sólo soporta HTTP/HTTPS: Esto es esencial para poder trabajar desde la oficina. Para ello tienen un ssh escuchando en el 443 del nombre ssh.github.com.

(más…)

I am playing arourd with puppet, a configuration management software.

We have both AIX and Linux systems, but I find puppet a little bit inmature for AIX.

Anyway, I think that it will be easy implement providers and recipes using the Exec providers on AIX. AIX has a consistent set of commands, and almost everything can be configured from command line. Most of commands have similar options, syntax and ouput.

Normally, all OS configuration items (users, groups, mountpoints) have a set of commands: ls, ch and mk. Most of then are based on attributes that can be changed/set and output of ls* commands usually can be printed in colons (-c option).

(más…)

BSoD for linux/unix/console

Publicado: junio 7, 2010 en fast-tip, humor, Personal, script
Etiquetas:, , ,

I do not remember where I got this, but it is fun. I will put it in the motd of my hosts on April Fools’ day.

This script will display on a console a windows like BSoD (Blue Script of Day):

                                    Linux ws         An exception 0E has occurred at 0028:C0018DBA in VxD IFSMGR(01) +        0000340A.  This was called from 0028:C0034118 in VxD NDIS(01) +        00000D7C.  It may be possible to continue normally.         *  Press any key to attempt to continue        *  Press CTRL+ALT+DEL to restart your computer. You will           lose any unsaved information in all applications                             Press Any key to continue. (más…)

AIX LDAP integration is not up to expectations. Its cache daemon, secldapclntd,
has a lot of problems:it often crashes, queries are slow, etc…

To mitigate problems, one workaround could be create the most important users locally,
using the KRB5files repository.

With this idea, this script will query a set of given groups from the AIX LDAP
registry using the AIX command line tools (lsuser, lsgroup), and it will create
them locally (mkgroup, mkuser).

To make it work, the host must be integrated with remote repository and must be able to resolve users and groups with LDAP method. You need LDAP method and KRB5files method configured. It can be easily changed to use other methods.

This script also supports nested groups from Active Directory.