Recent Updates RSS Hide threads | Keyboard Shortcuts

  • /proc and /sys tricks 

    ajdiaz 8:50 pm on 20 June 2009 Permalink | Reply

    Really the sysfs and /proc filesystems are worlds of magic and fantasy. Each day I discover a new trick using this filesystems. So, I decided to post a short summary of my favorites ones. Enjoy and feel free to add your tricks in comments, maybe we can a /proc and /sys knownledge database in a post :)

    1. Scanning for LUNs in attached FC
    echo "- - -" > /sys/class/scsi_host/hostX/scan

    2. CPU hotplug
    echp 0 > /sys/devices/system/cpu/cpuX/online

    Obviously when run echo 1, put the CPU online again

    3. Enable dmesg timestamp
    echo Y > /sys/modules/prinkt/parameters/time

    4. Restore a removed file when is still in use
    cat /proc/pid/fd/descriptor number > /tmp/myfile_restored

    5. Get the IO operations for a process:
    cat /proc/pid/io
    The syscr and the syscw are the accumulated read and write IO operations that process do where running.

    6. Increase size of IO scheduler queue:
    echo 10000 > /sys/block/device/queue/nr_request

    7. Get the current IO scheduler enabed to a specific device:
    cat /sys/block/device/queue/scheduler

    8. Get the threads of pdflush process which are running:
    cat /proc/sys/vm/nr_pdflush_threads

    9. Set the percentage threshold for memory to start to flushd data to disk:
    echo XX > /proc/sys/vm/dirty_background_ratio

    10. Set the sleep time for pdflush checking (in centisecs):
    echo XXX > /proc/sys/vm/dirty_writeback_centisecs

    11. Set the time to live for a data in buffer, when raises, data will commit to disk (in centisecs):
    echo XXX > /proc/sys/vm/dirty_expire_centisecs

     
  • Distributed tools 

    ajdiaz 7:29 pm on 3 April 2009 Permalink | Reply
    Tags: , , , config, gentoo

    For last months I needed to maintain a number of heterogeneous servers for mi work, I need to do some usually actions, like update a config file, restart a service, create local users etc.

    For this purposes there are a lot of applications, like dsh (or full csm), pysh, shmux and many others (only need to perform a search in google using phrase “distributed shell”). Unfortunately for me, I want a easy-to-parse solution, because I’ve a big (really big) number of servers, and I want a single cut-based/awk parsing, and also I need to do some actions as other users (like root, for example) via sudo. Althought many of the existants solutions offers me a subset of this features, I cannot found a complete solution. So I decided to create one :D

    You can find the code, and some packages in the dtools development site. I was use this solution in production environment from months with excelent results, and you can feel free to use.

    Of course, its free (of freedom) software, distributed under MIT license.

    Enjoy and remember: feedback are welcome ;)

     
  • bash4 is out! 

    ajdiaz 4:00 pm on 24 February 2009 Permalink | Reply
    Tags: , scripts

    Yerterday (monday 23 of Feb) the GNU team released a new version of bash.

    This new version contains a lot of interesting features, for example asociative arrays (yep!), autocd … and more!

    The notices was published in the bash mailing list.

     
  • netstrings implementation 

    ajdiaz 5:27 pm on 9 August 2008 Permalink | Reply
    Tags:

    This week I publish a tiny library to manage netstrings, as described in Bernstein’s paper. This library is wirten in C and provide a very basic implementation for small applications. I use this library in some projects for months, and now a publish the code over MIT License in launchpad.

    You can visit the page of the project in launchpad, or get the code via bzr:

    bzr get lp:netstr
    
     
  • pkgcore experience 

    ajdiaz 4:57 pm on 10 May 2008 Permalink | Reply
    Tags: gentoo systems pkgcore config

    Last days I reinstalled my Gentoo in my desktop computer, a Dell Dimension C521 using the new package handler pkgcore.

    The last months I’ve use official portage, and also new replacement called paludis. The main advantage of this one is being written entirely in C++, so paludis is very fast, but also you can detect some problems when you need to compile some “specials” packages, such qemu, which requires to be compiled with gcc3. If you compile paludis with another version of gcc, you can find a beauty error related to dynamic linking. Obviously, you can solve this problem by hand or with sonme tricks ;) , but I don’t like tricks in production machines.

    (More …)

     
    • hario 1:58 am on 15 May 2008 Permalink | Reply

      OMG! The configuration file is a good reason not to use pkgcore… yet :D

      I took a quick look at Pkgcore recently, but for now Paludis is doing a good job for me, so I will stick with it at least until a saner configuration syntax is added to Pkgcore – and no, the Portage compatibility is not enough for me, hehe.

    • Donnie Berkholz 8:45 am on 17 May 2008 Permalink | Reply

      I don’t even use pkgcore.conf, just the existing make.conf from portage.

    • ajdiaz 4:22 pm on 17 May 2008 Permalink | Reply

      Wel, I want to manage my repositories using pkgcore directly, without layman (only for testing :D ) and I think that need pkgcore.conf to do it. :(

      Am i right to do so?

    • Brian Harring 4:34 am on 6 December 2008 Permalink | Reply

      The ini format you used is pretty damn low level- frankly there isn’t any reason to drop down to that level unless you actually need to do something fairly crazy.

      Even then, you can intermix formats easily enough- in other words you could do an ini override of make.conf for example.

      Either way, ini is not the intended format for majority of users- it’s pretty much the intended format for if you need to do something fairly hanky (remote configuration, complex cache setups, etc).

  • Physics at MIT 

    ajdiaz 10:18 pm on 18 March 2008 Permalink | Reply
    Tags: education, , university

    I see in the “People of the Web” the last interview to the Professor Walter Lewin, at MIT. I have no comments about him and his classes, only two comments: 1) the great phrase: “Physics work!”, and 2) can other people regarded teacher? I don’t need to think more about it…

     
    • hario 2:36 am on 20 March 2008 Permalink | Reply

      Watching things like Walter Lewin classes makes one feel really sick about how teaching is done in Spain… we are many light years lagging behind the MIT, but you already know that, I guess :-(

  • Entanglement: the greatest mystery in physics 

    ajdiaz 1:09 am on 16 March 2008 Permalink | Reply
    Tags: book, , quantum

    Book cover

    Last days I was read the book titled “Entanglement: The greatest mystery in physics”, which is written by Ph.D. Amir D. Aczel, famous for his books about physics. This one is about quantum physics, particularly the entanglement phenomenon. The entanglement is an effect of quantum objects (if we can call them “objects”) consists in a relation between the quantum state of implicated objects, so when we alter one of them, the others “know” automagically the change, even though the objects are spatially separated.

    I’m surprised at how easy is to read, it’s funny and clearly (as much as quantum theory allows) and I recommended to people who are interesting in quantum theory and it’s history.

     
  • bash ini parser 

    ajdiaz 7:42 pm on 9 February 2008 Permalink | Reply
    Tags: ,

    In some situations i like to use INI files as configuration files, as python do. But bash do not provide a parser for these files, obviously you can use a awk code or a couple of sed calls, but if you are bash-priest and do not want to use nothing more, then you can try the following obscure code:

    cfg.parser () {
        IFS=$'\n' && ini=( $(<$1) )              # convert to line-array
        ini=( ${ini[*]//;*/} )                   # remove comments
        ini=( ${ini[*]/#[/\}$'\n'cfg.section.} ) # set section prefix
        ini=( ${ini[*]/%]/ \(} )                 # convert text2function (1)
        ini=( ${ini[*]/=/=\( } )                 # convert item to array
        ini=( ${ini[*]/%/ \)} )                  # close array parenthesis
        ini=( ${ini[*]/%\( \)/\(\) \{} )         # convert text2function (2)
        ini=( ${ini[*]/%\} \)/\}} )              # remove extra parenthesis
        ini[0]=''                                # remove first element
        ini[${#ini[*]} + 1]='}'                  # add the last brace
        eval "$(echo "${ini[*]}")"               # eval the result
    }
    

    And then you can parse your ini files as following:

    # parse the config file called 'myfile.ini', with the following
    # contents::
    #   [sec2]
    #   var2='something'
    cfg.parser 'myfile.ini'
    
    # enable section called 'sec2' (in the file [sec2]) for reading
    cfg.section.sec2
    
    # read the content of the variable called 'var2' (in the file
    # var2=XXX). If your var2 is an array, then you can use
    # ${var[index]}
    echo "$var2"
    

    Unfortunately, the cfg.parser() function do no support embedded spaces
    in section names… yet

     
    • Shawn 5:27 pm on 18 February 2008 Permalink | Reply

      Uh-oh:
      eval: line 2: syntax error near unexpected token `}’
      eval: line 2: `}’

      Hopefully this is just a typo in the code above as I’d be super happy to get this to work and I’m just not quite good enough to figure out what is wrong.

      Thanks for posting this!

    • ajdiaz 6:22 pm on 18 February 2008 Permalink | Reply

      Mmm, the code works for me :D

      There are my ‘myfile.ini’:

      [~]$ cat myfile.ini
      [sec2]
      var2=XXX

      And the result:
      [~]$ ./ini.sh
      XXX

      Maybe you need bash >= 3.0?, I’m using:

      [~]$ echo ${BASH_VERSION}
      3.2.17(1)-release

      If you want, send me the result of the command:

      $ bash -x ini.sh

      And I try to help you :)

      Thanks for your comments!! :D

    • Greg 1:58 pm on 3 March 2008 Permalink | Reply

      This script also doesn’t work if the ini file is not fully standard, like samba’s smb.conf. In that, there are some white spaces on both sides of the equation sign.

    • ajdiaz 10:50 pm on 8 March 2008 Permalink | Reply

      Sure, the script do not work fine with non stardard files, becaus the script parse file variables as bash variables and spaces are not allowed in bash variables. But, you are free to replace spaces with another character. :)

    • Tomas 2:24 pm on 29 April 2008 Permalink | Reply

      This works great!

      Do you have any functionality for updating the functions array dynamically without re-parsing the ini-file?

      Example: Add new section or add/update field within a section.

      I’m considering trying to code this, but it would of course save me lots of time if it already exists :-)

    • ajdiaz 3:34 pm on 1 May 2008 Permalink | Reply

      Sorry, I don’t code any functions to update the array. Really this hack do not use an array. We read the ini and create a couple of variables with the same name as variable in the file, and change the section creating a function called the section name. So if you need to add a new section, you might do:

      cfg.section.MYNEWSECTION ()
      {
      MYNEWVAR1=”MYNEWVALUE1″
      # any other variable
      }

      Now I’m working in a dump function to write the ini from specified functions.

    • Emanuele 3:16 pm on 8 May 2008 Permalink | Reply

      Hi,
      I’ve only copy the function in my bash script and I receive the following error:
      ‘cfg.parser’:not a valid identifier

      someone know why?

      BASH_VERSION -> 3.1.17(1)

    • ajdiaz 5:04 pm on 10 May 2008 Permalink | Reply

      Yes, probably your bash is running with POSIX option (check the value of $HELLOPTS), in this case the dot (.) is not a valid character for function name.

      If you really require a POSIX shell compatibility, you can replace cfg.parser and cfg.sections by cfg_parser and cfg_sections, the underline is always a valid character :D

    • Emanuele 10:00 am on 12 May 2008 Permalink | Reply

      Thanks! now I try!

    • Henning 2:37 pm on 20 June 2008 Permalink | Reply

      The source listing is somehow garbled. There’s something wrong with quotes (backticks?). The show up as a single character (&#8221 resp. &#8220=

    • ajdiaz 2:48 pm on 20 June 2008 Permalink | Reply

      Sure, the wordpress format is not very readable with selected theme, use the pastebin code in http://pastebin.com/f61ef4979 instead :)

    • Stephen 10:13 pm on 27 June 2008 Permalink | Reply

      The following will parse out a file that has spaces between the ‘=’ as well as both ; and # comments. Just thought I would share it.

      function cfg.parser ()
      {
      IFS=$’\n’ && ini=( $(<$1) ) # convert to line-array
      ini=( ${ini[*]//;*/} ) # remove comments ‘;’
      ini=( ${ini[*]//\#*/} ) # remove comments ‘#’
      ini=( ${ini[*]/\ =\ /=} ) # remove anything with a space around ‘ = ‘
      ini=( ${ini[*]/#[/\}$'\n'cfg.section.} ) # set section prefix
      ini=( ${ini[*]/%]/ \(} ) # convert text2function (1)
      ini=( ${ini[*]/=/=\( } ) # convert item to array
      ini=( ${ini[*]/%/ \)} ) # close array parenthesis
      ini=( ${ini[*]/%\( \)/\(\) \{} ) # convert text2function (2)
      ini=( ${ini[*]/%\} \)/\}} ) # remove extra parenthesis
      ini=( ${ini[*]/#\ */} ) # remove blank lines
      ini=( ${ini[*]/#\ */} ) # remove blank lines with tabs
      ini[0]=” # remove first element
      ini[${#ini[*]} + 1]=’}’ # add the last brace
      #printf “%s\n” ${ini[*]}
      eval “$(echo “${ini[*]}”)” # eval the result
      }

    • Lars 7:31 pm on 1 September 2008 Permalink | Reply

      Thanks for your script!

      Is there any chance to get the available variables of a section?

      For example: if I know that there’s a section [sec2], how can I print out the content of this section?

      I think I need something like this:

      for var in cfg.section.sec2; do
      print “variable: %s\n” $var;
      print “value: %s\n” $var[*];
      done

      but that doesn’t work for me.

    • ajdiaz 1:01 pm on 2 September 2008 Permalink | Reply

      declare -f cfg.section.sec2 | grep ‘=’ must be work Lars :)

    • frank 9:31 pm on 16 October 2008 Permalink | Reply

      Just a quick question… At one point I had this working, however, now all but the first section has been replaced by the number ‘1′. It’s replace right after this line:

      IFS=$’\n’ && ini=( $(<$1) )

      I’m using bash 3.0. Any thoughts?

      Thanks for the help.

    • Maciej Pasternacki 11:02 pm on 19 November 2008 Permalink | Reply

      One more fix: script changes $IFS and doesn’t restore previous value. Took me quote a moment of debugging, why the hell after reading config word splitting doesn’t work as they used to (e.g. LS=”ls -a”;$LS doesn’t work, because bash tries to find and execute “ls -a” command, including space and parameter). Fixed version:

      function cfg.parser ()
      {
      _old_IFS=”$IFS” # save $IFS
      IFS=$’\n’ && ini=( $(<$1) ) # convert to line-array
      ini=( ${ini[*]//;*/} ) # remove comments `;’
      ini=( ${ini[*]//\#*/} ) # remove comments `#’
      ini=( ${ini[*]/\ =\ /=} ) # remove anything with a space around `=`
      ini=( ${ini[*]/#[/\}$'\n'cfg.section.} ) # set section prefix
      ini=( ${ini[*]/%]/ \(} ) # convert text2function (1)
      ini=( ${ini[*]/=/=\( } ) # convert item to array
      ini=( ${ini[*]/%/ \)} ) # close array parenthesis
      ini=( ${ini[*]/%\( \)/\(\) \{} ) # convert text2function (2)
      ini=( ${ini[*]/%\} \)/\}} ) # remove extra parenthesis
      ini=( ${ini[*]/#\ */} ) # remove blank lines
      ini=( ${ini[*]/#\ */} ) # remove blank lines with tabs
      ini[0]=” # remove first element
      ini[${#ini[*]} + 1]=’}’ # add the last brace
      # printf “%s\n” ${ini[*]}
      eval “$(echo “${ini[*]}”)” # eval the result
      if [ -z "$_old_IFS" ] # restore old $IFS
      then unset IFS
      else IFS=”$_old_IFS”
      fi
      }

    • Brooks Moses 12:59 am on 16 January 2009 Permalink | Reply

      Maceij: Why not just use “local IFS” at the top of the function?

    • Piotr Gabryjeluk 12:57 pm on 23 February 2009 Permalink | Reply

      Yeah, should be

      local IFS instead of just IFS

      the rest of the script seemed a bit magic to me at first, but anyway nice peace of work. (I would suggest not using a dot in name of function).

    • Markus E. 1:34 pm on 7 May 2009 Permalink | Reply

      nice work!

      the error described in first post occurs if a section does not contain values.
      this can be solved by adding any command between the brackets which are evaluated by eval().

      i think, there could be a better solution, but following works;
      replace:
      ni[${#ini[*]} + 1]=’}’ # add the last brace

      with something like:
      ini[${#ini[*]} + 1]=’echo -n; }’

    • neofutur 5:17 am on 20 May 2009 Permalink | Reply

      useful post, but the wordpress format makes it a hell to use it.

      what about packaging this in a .tar.gz and give the download link ?

    • bld 2:56 pm on 21 May 2009 Permalink | Reply

      This Bash ini parser may be easier to cut & paste:

      http://codesnippets.joyent.com/posts/show/2060

      cheers,

      bld

    • nic 11:04 pm on 17 August 2009 Permalink | Reply

      Hello!

      I tried the wonderful script and I’m even starting to understand what it does. I would like to modify the two areas where the text is converted to function and key names to do this: uppercase the names and in the case of the section name replace blank characters with underscore. I don’t know how to do this, especially since the script runs the regex for the whole file in one swing (IE not line by line).

      Best regards

      • ajdiaz 8:11 pm on 19 August 2009 Permalink | Reply

        Hi!

        To put the ini contents in uppercase is easy, just change the line:
        IFS=$'\n' && ini=( $(<$1) )
        by the line:
        IFS=$'\n' && ini=( $(tr a-z A-Z <$1) )

        This will be work ok to you. About the spaces in sections, there are no a easy way to do this AFAIK.

        I hope to help you a bit.

        Regards,
        Andres

  • bashdoc: just another documentation tool 

    ajdiaz 11:27 pm on 8 February 2008 Permalink | Reply
    Tags:

    bashdoc is a small utility to make documentation automatically from bash scripts, using awk to frontend parser (and you can add your own frontends in awk language), and reStructuredText as backend parser. bashdoc parse the object script (using awk) and create an intermediate documentation in RST, which is parsed in next step using RST backend.

    The fronted allows you to parse more complicated scripts (or other than bash scripts) and the backend allows you to make the output in different formats.

    You can download the source code from launchpad project page or using bzr version control system:

    $ bzr get lp:bashdoc

     
  • ajdiaz 10:37 pm on 1 February 2008 Permalink | Reply
    Tags: , floss,

    comida is a bash script designed to create a FTP (or HTTP, or rsync or…) mirror and maintain the archive synchronized with the source. I design this tool when I was working at the Free Software Office (OSL) in the University of A Corunna, and decided to create a new mirror in Spain which host a lot of free software projects. But, we wanted a single tool to manage all mirrors in the archive, with full integration in our web page (yes, we wanted on-the-fly status page).

    You can download the source code from launchpad project page or using bzr version control system:

    $ bzr get lp:comida
    
     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
esc
cancel