/proc and /sys tricks

Really the sysfs and /proc filesystems are worlds of magic and fantasy. Each day I discover a new trick using this filesystems. So, I decided to post a short summary of my favorites ones. Enjoy and feel free to add your tricks in comments, maybe we can a /proc and /sys knownledge database in a post

1. Scanning for LUNs in attached FC
echo "- - -" > /sys/class/scsi_host/hostX/scan

2. CPU hotplug
echp 0 > /sys/devices/system/cpu/cpuX/online

Obviously when run echo 1, put the CPU online again

3. Enable dmesg timestamp
echo Y > /sys/modules/prinkt/parameters/time

4. Restore a removed file when is still in use
cat /proc/pid/fd/descriptor number > /tmp/myfile_restored

5. Get the IO operations for a process:
cat /proc/pid/io
The syscr and the syscw are the accumulated read and write IO operations that process do where running.

6. Increase size of IO scheduler queue:
echo 10000 > /sys/block/device/queue/nr_request

7. Get the current IO scheduler enabed to a specific device:
cat /sys/block/device/queue/scheduler

8. Get the threads of pdflush process which are running:
cat /proc/sys/vm/nr_pdflush_threads

9. Set the percentage threshold for memory to start to flushd data to disk:
echo XX > /proc/sys/vm/dirty_background_ratio

10. Set the sleep time for pdflush checking (in centisecs):
echo XXX > /proc/sys/vm/dirty_writeback_centisecs

11. Set the time to live for a data in buffer, when raises, data will commit to disk (in centisecs):
echo XXX > /proc/sys/vm/dirty_expire_centisecs

Distributed tools

For last months I needed to maintain a number of heterogeneous servers for mi work, I need to do some usually actions, like update a config file, restart a service, create local users etc.

For this purposes there are a lot of applications, like dsh (or full csm), pysh, shmux and many others (only need to perform a search in google using phrase “distributed shell”). Unfortunately for me, I want a easy-to-parse solution, because I’ve a big (really big) number of servers, and I want a single cut-based/awk parsing, and also I need to do some actions as other users (like root, for example) via sudo. Althought many of the existants solutions offers me a subset of this features, I cannot found a complete solution. So I decided to create one

You can find the code, and some packages in the dtools development site. I was use this solution in production environment from months with excelent results, and you can feel free to use.

Of course, its free (of freedom) software, distributed under MIT license.

Enjoy and remember: feedback are welcome

bash4 is out!

Yerterday (monday 23 of Feb) the GNU team released a new version of bash.

This new version contains a lot of interesting features, for example asociative arrays (yep!), autocd … and more!

The notices was published in the bash mailing list.

netstrings implementation

This week I publish a tiny library to manage netstrings, as described in Bernstein’s paper. This library is wirten in C and provide a very basic implementation for small applications. I use this library in some projects for months, and now a publish the code over MIT License in launchpad.

You can visit the page of the project in launchpad, or get the code via bzr:

bzr get lp:netstr

pkgcore experience

Last days I reinstalled my Gentoo in my desktop computer, a Dell Dimension C521 using the new package handler pkgcore.

The last months I’ve use official portage, and also new replacement called paludis. The main advantage of this one is being written entirely in C++, so paludis is very fast, but also you can detect some problems when you need to compile some “specials” packages, such qemu, which requires to be compiled with gcc3. If you compile paludis with another version of gcc, you can find a beauty error related to dynamic linking. Obviously, you can solve this problem by hand or with sonme tricks , but I don’t like tricks in production machines.

Physics at MIT

I see in the “People of the Web” the last interview to the Professor Walter Lewin, at MIT. I have no comments about him and his classes, only two comments: 1) the great phrase: “Physics work!”, and 2) can other people regarded teacher? I don’t need to think more about it…

Entanglement: the greatest mystery in physics

Last days I was read the book titled “Entanglement: The greatest mystery in physics”, which is written by Ph.D. Amir D. Aczel, famous for his books about physics. This one is about quantum physics, particularly the entanglement phenomenon. The entanglement is an effect of quantum objects (if we can call them “objects”) consists in a relation between the quantum state of implicated objects, so when we alter one of them, the others “know” automagically the change, even though the objects are spatially separated.

I’m surprised at how easy is to read, it’s funny and clearly (as much as quantum theory allows) and I recommended to people who are interesting in quantum theory and it’s history.

bash ini parser

In some situations i like to use INI files as configuration files, as python do. But bash do not provide a parser for these files, obviously you can use a awk code or a couple of sed calls, but if you are bash-priest and do not want to use nothing more, then you can try the following obscure code:

cfg.parser () {
    IFS=$'\n' && ini=( $(<$1) )              # convert to line-array
    ini=( ${ini[*]//;*/} )                   # remove comments
    ini=( ${ini[*]/#[/\}$'\n'cfg.section.} ) # set section prefix
    ini=( ${ini[*]/%]/ \(} )                 # convert text2function (1)
    ini=( ${ini[*]/=/=\( } )                 # convert item to array
    ini=( ${ini[*]/%/ \)} )                  # close array parenthesis
    ini=( ${ini[*]/%\( \)/\(\) \{} )         # convert text2function (2)
    ini=( ${ini[*]/%\} \)/\}} )              # remove extra parenthesis
    ini[0]=''                                # remove first element
    ini[${#ini[*]} + 1]='}'                  # add the last brace
    eval "$(echo "${ini[*]}")"               # eval the result
}

And then you can parse your ini files as following:

# parse the config file called 'myfile.ini', with the following
# contents::
#   [sec2]
#   var2='something'
cfg.parser 'myfile.ini'

# enable section called 'sec2' (in the file [sec2]) for reading
cfg.section.sec2

# read the content of the variable called 'var2' (in the file
# var2=XXX). If your var2 is an array, then you can use
# ${var[index]}
echo "$var2"

Unfortunately, the cfg.parser() function do no support embedded spaces
in section names… yet

bashdoc: just another documentation tool

bashdoc is a small utility to make documentation automatically from bash scripts, using awk to frontend parser (and you can add your own frontends in awk language), and reStructuredText as backend parser. bashdoc parse the object script (using awk) and create an intermediate documentation in RST, which is parsed in next step using RST backend.

The fronted allows you to parse more complicated scripts (or other than bash scripts) and the backend allows you to make the output in different formats.

You can download the source code from launchpad project page or using bzr version control system:

$ bzr get lp:bashdoc

comida is a bash script designed to create a FTP (or HTTP, or rsync or…) mirror and maintain the archive synchronized with the source. I design this tool when I was working at the Free Software Office (OSL) in the University of A Corunna, and decided to create a new mirror in Spain which host a lot of free software projects. But, we wanted a single tool to manage all mirrors in the archive, with full integration in our web page (yes, we wanted on-the-fly status page).

You can download the source code from launchpad project page or using bzr version control system:

$ bzr get lp:comida