Since I had more than one hundred of hosts, and this number could be grow up in short, and I’ve a number of users which need to access to all hosts, I need to think in a way to easy user management. Actually user management is, in my opinion a pain in the butt. If you ve a central user directory, you need to deal with a big and fat single point of failure, so you need to create some kind of HA service for this directory. And if you ve systems around the world, then you need to replicate the user data in different directories and keep them synchronized. If it is not the hell, it must be very similar.

Dealing with user managemet is a royal hassle for system administrators in every place, but in the cloud (i.e. a number of hosts distributed around the world), it’s also a punishment. So, I need to solve (almost in part) this problem before moving forward in my deployment. I do not need a full user management really, just a basic UID mapping and a way to authenticate users (for which I could use the old-and-friendly authorized_keys).

So, How can I manage a big number of users in a single way, and to be effective in a distributed environment? That’s not a simple questions, and of course each implementation has its own solution, from authentication services to suites of scripts. Anyway, I was looking for a simple to manage ones, cause of I was the responsible to manage the entire environment and I’m too lazy too

Thinking about the problem, I imagine a system without any user, let’s imaging that there are just one user, and any other user is just an alias for the first one. It could be easy to manage, because we only need one UID, but we need to solve the alias mapping.
Here is when libnss_map join into the game. The libnss_map is a library designed to be used with GNU NSS service. The NSS allows the system to get user credentials from many sources, which can be configured easy from the /etc/nsswitch.conf file.

For example, we can configure our system in the following way:

passwd:      files map
shadow:      files map
group:       files map

So, for each user to get credentials NSS will lookup in standard files first, and then using the map module (libnss_map).The map module works as the flow diagram shows.

As you can see in the diagram there are two major steps in lookup. The first one is the responsible to map an user to a virtual one. The virtual user is static, and it’s defined in /etc/nssmap.conf. This file has the same syntax like passwd does. For example:

virtual:x:10000:10000:,,,:/home/:/secure_shell

Which means that any user who does not exists in /etc/passwd will be mapped into this one, with UID 10000.

Okay, sounds good, but there are a lot questions yet. What about the password? What about the home dir?

Well, I do not find a good solution for password, so nssmap will return a masked password (account is enabled, but password will be unpredictable), and I authenticate the user using other methods via PAM, or public keys via SSH.

Home directory is easier. The home directory field in the user definition (inside /etc/nssmap.conf file) is used as prefix, and it will be completed with the user name (the name of the user which is intended to login, not the virtual one). So, for example, for the hypothetical user “sample”, the effective home directory will be “/home/sample”, because “/home/” is the prefix. Please note that the end slash is mandatory in current implementation.

Finally I need to solve another big problem: if two users has the same UID then both can change the same files, or delete the files of other “virtual” user. How can we solve it? There are not single answer, not easy afterall. In my case I use special shell, which
ensure that the user cannot remove, touch or even read files in any path into /home except his own home directory, but it’s not a full solution yet.

Here is an example using nss map:

host ~ # sudo su - test
No directory, logging in with HOME=/
test@host / $ id
uid=10000(virtual) gid=10000(virtual) groups=10000(virtual)

In the meanwhile, a basic code is available in my github, and I still researching in this kind of authorization. Keep in touch and enjoy! An of course, feedback is welcome

Update: Óscar García gently published a similar spanish version of this article.

Couple of weeks ago I received my dreamplug, from NewIT. Though I knew some software troubles in device, I remains hopeful. However when I unpacked and powered on the plug I got a number of problems. This is the history about these problems and their solutions. I hope that my experience can be useful for anyone who take the same model and have the same problems. Afterall, the NewIT forum has a number of posts about these and other problems, so reading it is recommended.

The first problem that I found was a bad partition table, which is also a well-known bug in NewIT forums; and the second one is the Ubuntu installation which is not very tiny by default Fortunately, both problems are easy to solve.

These are the steps that I followed to “upgrade” my dreamplug software to a cooler one. Please note that the Globalscale guys do not support other system images and installations AFAIK, so perform this changes under your own responsibility. You are warned!.

Anyway, that is the recipe:

STEP ONE: Creating a bootable USB

The first step in our process is to create a pseudo-bootable USB. The “pseudo” prefix is here because we don’t need really a bootable USB, but a USB storage with the contents of the image inside. We will use the same uImage provided in the original dreamplug or, of course, you can change the kernel image whenever you want.

For this USB image you can use a factory one, debian image provided by Globalscale guys, or my own debian image (which is my favourite one, of course). You also need to format your USB with ext2, ext3 or ext4 filesystem (the current Dreamplug kernel do not support much filesystems), and then unpack the image into it. It’s very simple step, for example, if you have plowshare installed, you can follow these steps:

$ plowdown http://www.megaupload.com/?d=R59E1DU3
$ mkfs.ext4 /dev/sdñ # or your USB unit
$ mount -t ext4 /dev/sdñ /mnt
$ tar -xvpzf dantzig-dreamplug-image.tar.gz -C /mnt
$ cp dantzig-dreamplug-image.tar.gz /mnt/root # only required for step 4.

In this example I suppose that device /dev/sdñ exists and it’s the USB device.

STEP TWO: Boot from USB

The next step is to boot from the USB that we make in the last step. To perform well this step we also need to have attached a JTAG unit. Without the JTAG we cannot control the boot process and we cannot force to u-boot to use the USB. The idea is to boot from the factory kernel image, but setting the root filesystem to the USB.

In other plug models, like Guruplug, you can change the boot environment from the OS, so you do not require the JTAG (if everything works fine), but AFAIK, Dreamplug has u-boot code in a different flash memory which cannot be modified from the OS. If anyone knows a way to do this, please don’t keep it in secret

When system starts in JTAG console you can see the following lines:

88E1121 Initialized on egiga1
Hit any key to stop autoboot: 3

Then you can press any key to enter in the u-boot prompt:

Marvell>>

Then change the environment to point the root filesystem to the USB:

Marvell>> setenv x_bootargs_root root=/dev/sdc1 rootdelay=15
Marvell>> boot

The rootdelay paramenter is required to wait while the USB bus is initializing after kernel loading. While kernel bus scan is asynchronous, you need to wait until the USB one starts successfully. You cannot control this time and it’s depends on many factor, but usually a 15 sec value is time enough.

Please note that we never commit the changes into the environment, we boot the kernel with the new one, but not save this environment because we do not need to start from the USB anymore.

STEP THREE: Rebuilding partitions.

Once the system start (will take ~30s), you can login with the root password. The password depends on the image that you use in first step, but the password nosoup4u is default one for my image and for factory ones too.

Once you get a prompt, is time to take a look into partition table:

# fdisk -l /dev/sda
Disk /dev/sda: 1967 MB, 1967128576 bytes
61 heads, 62 sectors/track, 1015 cylinders
Units = cylinders of 3782 * 512 = 1936384 bytes
Disk identifier: 0xabcdef

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1016     1920993+   6  FAT16
Partition 1 has different physical/logical endings:
     phys=(54, 61, 61) logical=(1015, 54, 24)
/dev/sdb2              56        1021     1826706   83  Linux

As you can see we’ve a problem with partitions. The first one is a FAT16 partition, which store the kernel in uImage format, which is loaded by u-boot when booting, but as you see in the previously output, the size is not correct. The partition (as documentation said) might have a size of 16M, that is 55 cylinders, so we need to remove the partition and create a new one:

# fdisk /dev/sda
Command (m for help): d
Partition number (1-4): 1
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1015, default 1): 1
Last cylinder, +cylinders or +size{K,M,G} (1-55, default 55): 55

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): 6
Changed system type of partition 1 to 6 (FAT16)

Command (m for help): w
The partition table has been altered!

Syncing disks.

Now the first partition has the correct size, but unfortunately, the second partition is incorrect too. It starts at cylinder 56, which is correct, but ends beyond the end of disk. We need to correct the size of this partition too, in the same way as we do in the last step:

# fdisk /dev/sda
Command (m for help): d
Partition number (1-4): 2
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (56-1015, default 56): [enter]
Using default value 56
Last cylinder, +cylinders or +size{K,M,G} (56-1015, default 1015): [enter]
Using default value 1015

Command (m for help): w
The partition table has been altered!

Syncing disks.

And that’s all. At this point we have the correct partition size in both partitions, and the system might work fine. We only need to reboot the system, sacrify a goat and pray. But maybe you want to delete Ubuntu and install other distribution, let’s say Debian or even better Gentoo. As my boot image is Debian based, it’s easy to delete Ubuntu and install Debian Squeeze (which is the image base)…

STEP FOUR: Debianize

If you remember the step one, we also copied the dantzig-dreamplug-image.tar.gz image into /root. Now we will use this targz to unpack it into sdb disk.

Let’s suppose that you are booting from USB image and login the system and get a root console (steps 1,2),then:

# umount /dev/sda2 # Puesto que ha sido montado automáticamente por usbmount
# mkfs.ext4 /dev/sda2 # Formateamos con ext4
# e2label /dev/sda2 root # Etiquetamos la partición como root (opcional)
# mount -t ext4 /dev/sda2 /mnt
# tar xvzpf /dreamplug_debian_v0.2.tar.gz -C /mnt
# sync && sync && sync
# umount /mnt

At that point we must have a Debian distro installed in sdb2, and in next reboot the new system will be booted.

Maybe you need to know a bit more about dantzig-dreamplug-image.tar.gz. This is the image which I created to my plugs (dreamplug and guruplug) and it works in production for connectical.com site for months. This image uses runit as init process and a lot of features and geek services

Enjoy!

In the last months, dtools becomes in an useful tool for me. I use dtools everyday for system administration in large distributed networks. So I decided to improve some functionalities and also test and retest current features, so in a couple of months I expect to launch a new release of dtools.

In the meantime, I still working in whistler, a XMPP bot for MUC rooms, I hope that SleekXMPP library, which is the XMPP engine used by whistler become to release early. At that point we must remove any dependency with the old xmpppy in the code and we will ready to release the version 2.0 of whistler.

For example you can remove the left zeroes in a decimal number without require any external utility or print formats, let’s suppose that you want to strip zeroes from the number 007, which is stored in bond variable.

$ echo $bond
007
$ let nozeros=10#$bond
$ echo $nozeros
7

In many forums and mailing list, people need to use ugly sed expressions, or awk invokation, but (with bash) it’s just simply

Using the same trick, you can perform a base conversions, for example:

$ let i=0x10
$ echo $i
16
$ let i=2#10000
$ echo $i
16

Or create an easy number checking:

$ is_decimal () { let i=10#$1 2>/dev/null; }
$ is_decimal 'a' || echo Nop
Nop
$ is_decimal 56 && echo 'Yep'
Yep

Enjoy!

Whistler is intended to manage the connectical MUC room, and only basic functionalities are provided. Obviously it is under heavy development yet.

The code is publicy available on github whistler repository, and you can clone as usually:

$ git clone git://github.com/ajdiaz/whistler

You require xmppy to work with whistler and python >= 2.5. In few days I will publish the projecti into pypi too.

Enjoy and remember, any feedback is welcome

Now I will try to include my repo in unavailable-unofficial list… In the meanwhile, here is the configuration that you need for paludis:

location = ${ROOT}/var/db/paludis/repositories/ajdiaz-exheres sync = git://github.com/ajdiaz/ajdiaz-exheres format = e

Enjoy!

The first one is moving my dot files to git (and probably pushed them to github), and the second one is to create a hook for git to update my dot files. I known that there are a lot of similar solutions, one more complex, other more easy, but this is mine

So, I created a post-commit hook script for git, which perform the modifications that I need. Now I just only do this steps:

1. Create a new git repo:

mkdir mydots_repo
cd my_dots_repo && git init

2. Put the hook:

wget -O .git/hooks/post-commit  http://2tu.us/2scm 
chmod 755 .git/hooks/post-commit

3. Copy old files:

cp ~/old/bzr_repo/* .
git add *
 

4. Commit and recreate links:

git commit -a -m'initial import'

And it’s works

We are discussing now about how distribute the infraestructure and how to keep a number of copies in remote locations up-to-date, we are exploring solutions like elliptics or some similar.

In the meanwhile I created my github account to still my projects under development, and also to have a backup of some projects that I really use everyday.

Enjoy!

Features

• Fully written in bash, no third party software required (except ssh, obviously).
• Based in module architecture, easy to extend.
• Full integration with ssh.
• Easy to group hosts by tags or search by regular expression.
• Manage of ssh hosts
• Parseable output, but human-readable
• Thinking in system admin, no special development skills required to extend the software.

Short Example

$ dt tag:linux ssh date
okay::dt:ssh:myhostlinux1.domain:Mon Nov 16 23:54:04 CET 2009
okay::dt:ssh:myhostlinux3.domain:Mon Nov 16 23:54:04 CET 2009
okay::dt:ssh:myhostlinux2.domain:Mon Nov 16 23:54:04 CET 2009

As usual, you can download the code from the project page, or if you wish you can download the code via git:

git clone git://git.connectical.com/ajdiaz/dtools

Enjoy!

But (and it’s a big “but” for me) I really dislike the color scheme that use by default. htop comming with five color schemes, but I cannot find any beautifull one (from my personal point of view, of course), so I decided to make a new schema. I called “blueweb” theme (dont’ ask) . And here is the result:

htop with blueweb theme

You can download the patch file for the htop source code. And yes, unfortunately you need to patch the code.

Now my htop looks nice

Enjoy!