Setting the LS_COLORS environment variable to enable colorization in the Tree command

I've come to love the use of the Tree command to print a directory listing in hierarchical form.  I often will use it for documentation purposes such as illustrating how a project hierarchy is laid out, identifying where a specific file is located within the project directory, etc.

I will often use Tree to explore a directory that I am not familiar with as well.  For example, if I download an archive file, it can be much quicker to use Tree after archive is uncompressed, than trying to click back and forth through folders to understand the hierarchy and contents.

However, the Tree command requires the LS_COLORS environment variable be set, which is not typically set by default.  Since most of use simply create an alias such as alias ls='ls -color' or, in my case ll='ls -al --color', which provides colorization to directory listings without using LS_COLORS, we don't have much reason for setting the environment variable.

I tracked down a few blog posts and other sources on the interwebs that cover the setting of LS_COLORS, but none of them really stated how to set it, so I'm capturing it here in hopes that it will be useful for others.

DIR_COLORS

Ultimately, most of the posts I found on the interwebs deal with *changing* the default colors, but I simply wanted the default colors set in the environment variable.  For this, we turn to DIR_COLORS.

Per the man page for DIR_COLORS, the default colors "database" resides under /etc/DIR_COLORS.

This file is simply a listing of the default color settings for the types of files that the ls command displays in the terminal.  

The following is a snippet of the section we're interested in.

$ cat /etc/DIR_COLORS
# Coreutils version coreutils-8.23
...

# Below are the color init strings for the basic file types. A color init
# string consists of one or more of the following numeric codes:
# Attribute codes:
# 00=none 01=bold 04=underscore 05=blink 07=reverse 08=concealed
# Text color codes:
# 30=black 31=red 32=green 33=yellow 34=blue 35=magenta 36=cyan 37=white
# Background color codes:
# 40=black 41=red 42=green 43=yellow 44=blue 45=magenta 46=cyan 47=white
#NORMAL 00 # no color code at all
#FILE 00 # regular file: use no color at all
RESET 0 # reset to "normal" color
DIR 01;34 # directory
LINK 01;36 # symbolic link. (If you set this to 'target' instead of a
# numerical value, the color is as for the file pointed to.)
MULTIHARDLINK 00 # regular file with more than one link
FIFO 40;33 # pipe
SOCK 01;35 # socket
DOOR 01;35 # door
BLK 40;33;01 # block device driver
CHR 40;33;01 # character device driver
ORPHAN 40;31;01 # symlink to nonexistent file, or non-stat'able file
SETUID 37;41 # file that is setuid (u+s)
SETGID 30;43 # file that is setgid (g+s)
CAPABILITY 30;41 # file with capability
STICKY_OTHER_WRITABLE 30;42 # dir that is sticky and other-writable (+t,o+w)
OTHER_WRITABLE 34;42 # dir that is other-writable (o+w) and not sticky
STICKY 37;44 # dir with the sticky bit set (+t) and not other-writable

# This is for files with execute permission:
EXEC 01;32

 In the man page for DIR_COLORS, it states that running eval `dircolors some_path/dir_colors` will set the environment variable.  

Alternatively, running `dircolors --print-database ~/.dircolors` will make a copy of the file from /etc/DIR_COLORS in ~/.dircolors where you can override the default colors to your liking.

The color codes can be found here.  Thanks Bartman!

Setting LS_COLORS

Within Cygwin, the LS_COLORS environment variable can be set by simply running `dircolors`:

$ dircolors

LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:';

export LS_COLORS

This will use the system defaults from the /etc/DIR_COLORS file. 

If you created a local copy to override the default color scheme, you need to identify the file to use by running `dircolors ~/.dircolors`.

Viola!  It's as simple as that and now Tree looks great and makes directory hierarchy illustration and understanding easier and more colorful!

$ tree /etc/X11
/etc/X11
├── app-defaults
│ ├── KOI8RXTerm
│ ├── KOI8RXTerm-color
│ ├── UXTerm
│ ├── UXTerm-color
│ ├── XTerm
│ └── XTerm-color
├── system.XWinrc
├── xinit
│ ├── startxwinrc
│ ├── Xclients
│ ├── Xclients.d
│ │ └── Xclients.xinit-compat.sh
│ ├── xinitrc
│ ├── xinitrc.d
│ │ ├── 00-start-message-bus.sh
│ │ ├── localuser.sh
│ │ └── xdg-user-dirs.sh
│ ├── xinitrc-common
│ └── Xsession
├── Xmodmap
└── Xresources

4 directories, 18 files

Installing Python3, NumPy, SciPy and MatPlotLib under Cygwin on Windows 7

I had a working Cygwin environment with Python 2.7 installed, but I needed to install Python3, NumPy, SciPy, and MatPlotLib, which brought about a whole slew of issues.  I had to scour the web for the following solutions, but I was not able to find a single

I'm a command line junky, so I had already installed apt-cyg.  The following will outline the steps I had to do to get the packages identified above installed.

Install Python3

Installing Python3 is easy enough.

$ apt-cyg install python3

Now, before we attempt to install NumPy, SciPy, and MatPlotLib, we need to install a bunch of other dependencies, if they aren't already installed.  These include python3-setuptools, liblapack-devel, libopenblas, gcc-fortran, gcc-g++, python-gtk2, and libfreetype-devel.

$ apt-cyg install python3-setuptools liblapack-devel libopenblas gcc-fortran gcc-g++ python-gtk2 libfreetype-devel

Next, we need to create a symbolic link of /usr/include/locale.h to /usr/include/xlocale.h:

$ ln -s /usr/include/locale.h /usr/include/xlocale.h

Finally, we need to install pip and upgrade it.  Note, if you haven't updated your PATH environment variable the following command may not work until you do that or open a new Cygwin terminal.

$ python3 -m pip install --upgrade pip

Now we should be ready to install NumPy, SciPy, and MatPlotLib using pip.

Install NumPy, SciPy, and MatPlotLib

$ pip install numpy$ pip install scipy$ pip install matplotlib

 That should be it.

Automating ModelSim/QuestaSim Regressions

Implementing a makefile driven regression setup to automate the execution of testcases of varying length requires some method of gracefully ending the simulation. Verilog supports the use of $finish to signal the end of the simulation. VHDL 2008 introduces the 'env' package to the 'std' library that includes a method similar to Verilog:

library std;use std.env.all;.......stop(0);--orfinish(0);-- For both STOP and FINISH the STATUS values are those used-- in the Verilog $finish task-- 0 prints nothing-- 1 prints simulation time and location-- 2 prints simulation time, location, and statistics about-- the memory and CPU times used in simulation-- Other STATUS values are interpreted as 0.

If the project requires VHDL 93, then the following method can be used instead.

1. Add a signal to the testbench top level that can be asserted by the testcase to indicate the end of the simulation.
2. Add the following to the do-file used to execute the simulation:

## Initialize broken variable to 0set broken 0set finished 0## assert statments with a severity of failure cause breakpoints## when a breakpoint is encountered the following routine runs## which sets the broken variable.onbreak {  if {$finished} {    resume  } else {    set broken 1    resume  }}when -label end_of_simulation {sim_done_s == '1'} {echo "simulation finished"; set finished 1; stop;}## run until breakrun -all## if the sim was brokenif {$broken } {puts "Test Failed to Complete"} else {puts "Test Successfully Completed"}quit -f

In this example, I added a signal to my testbench called "sim_done_s" which is initialized to '0'. At the end of the testcase, sim_done_s is assigned to '1'.

Alternatively, the more common approach is to simply apply an assertion of the appropriate level to cause the simulator to break.  This is set in the modelsim.ini file and defaults to a level of Failure.  I personally don't like this method because I don't want to see the word Failure in my transcript if there isn't actually a failure.  This way I can automate the regressions and run a grep on the logs for the word Failure or Error.

Command Line Fu

I had an SVN Add crash on me and now the entire folder is locked such that any attempt to perform anything on that folder errors out. The Adds that did complete appear to be only locally and not pushed to the server, so I'm going to try to blow away all of the .svn folders locally and try adding again..... but that's a lot of manual deleting in Winblows, so here's some command-line-fu in Cygwin.

$ find . -iname *.svn | xargs -r -n 20 rm -rf

The 'find' stuff is obvious. The rest pipes the output from find into xargs which creates a list of 20 and executes 'rm -rf' on that list.  The list size would obviously need to be adjusted to the expected number of hits.  Sweet.

On another note, I recently had to resurrect an old project originally developed on Solaris.  This particular project environment used several layers of Makefiles that called several different Perl scripts.  The Perl scripts all pointed at the wrong path for the Perl executable in my Linux environment, so I had two options: 1) create a symbolic link in the directory the scripts pointed to (hack!) or 2) modify each of the scripts to point to the correct directory.  The first is a hack, but the second takes quite a bit more time and when it comes to monotonous tasks like that, I'm fairly lazy and prefer an automated method.  Hence the following:

$ grep –rl ‘/usr/local/bin/perl’ *.pl | xargs sed -i  ‘s/\/usr\/local\/bin\/perl/\/usr\/bin\/perl/g’

The '-r' option tells grep to operate recursively through subdirectories and the '-l' option tells grep to return only the filename of the files containing matches.  The output is piped to xargs and a simple sed switch is executed on each of the files.  Brilliant!  

Installing Old Libraries on Ubuntu

As a Design Services company, we quite often will have to support an old project for a customer that was implemented in an old version of the tools.  This typically requires us to install older versions of the tools when the customer doesn't want to update the design to the new tools for some reason.

I recently had to support a project that was originally implemented in Xilinx ISE v9.1.  Installing this on Ubuntu Server 11.04 64-bit required some extra effort.

When attempting to install Xilinx ISE v9.1 tools on the Builder box (Ubuntu Server 11.04), I received the following error when attempting to run the installer.

/home/builder/ISE_DVD_J.30.5.0/bin/lin64/setup: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory

So what’s installed?

$ dpkg --get-selections | grep libstdc++
libstdc++6

Ah. Okay, so we need to install libstdc++5.

$ sudo apt-get install libstdc++5
...
$ dpkg --get-selections | grep libstdc++
libstdc++5 install
libstdc++6 install

$ dpkg -L libstdc++5
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libstdc++5
/usr/share/doc/libstdc++5/TODO.Debian
/usr/share/doc/libstdc++5/copyright
/usr/share/doc/libstdc++5/changelog.Debian.gz
/usr/share/doc/libstdc++5/README.Debian
/usr/lib
/usr/lib/libstdc++.so.5.0.7
/usr/lib/libstdc++.so.5

$ ll /usr/lib/libstdc++*
lrwxrwxrwx 1 root root 18 2013-03-26 14:03 /usr/lib/libstdc++.so.5 -> libstdc++.so.5.0.7
-rw-r--r-- 1 root root 829320 2011-02-19 11:01 /usr/lib/libstdc++.so.5.0.7

Works!

One thing to make note of – if the applications are old enough to have this problem, they will likely have problems with other libraries such as libXm from the libmotif package. In this case, the compxlib tool required libXm.so.3 from libmotif3, but later versions of Ubuntu have libmotif4 in the repos. The easiest workaround is to install libmotif4 and sym link the library:

$ sudo apt-get install libmotif4
$ ln -s /usr/lib/libXm.so.4 /usr/lib/libXm.so.3

Automating Timestamp Updates at Build Time

In the past, I've used a simple makefile to update the revision register value to the current system time using a simple sed command as follows:

buildStamp=$(shell date +%m%d%y%H)
sed 's/ABCDEF01/$(buildStamp)/' ../ip/common/buildver_pkg.vhd > $(localTemp)/buildver_pkg.vhd

This works well when we're using a strictly command line build flow. I typically have the make file copy all of the source to a timestamped sub-directory and make this update at the time of the copy. This preserves the pedigree of the build and allows me to rerun builds from that directory without updating the revision register every time.

However, a problem arises when the Quartus II GUI is used because the revision register won't be updated. The following is one method for making the GUI update the revision register.

Add a script like this:

##############################################################################################
#
# This script copies the buildver_pkg.vhd file to the local build directory, and updates the
# placeholder string to the current system time.
#
##############################################################################################
proc copyfile { args } {
set timestamp [clock format [clock seconds] -format {%y%m%d%H}]
# Year 0-99, 7 bits
# Month 1-12, 4 bits
# Day 1-31, 5 bits
# Hour 0-23, 5 bits
set ifp [open "../ip/common/buildver_pkg.vhd" r]
set ofp [open "buildver_pkg.vhd" w]
while { [gets $ifp line] >= 0 } {
regsub -all "ABCDEF01" $line $timestamp line
puts $ofp $line
}
close $ifp
close $ofp
}
copyfile

Now, add the following line to the QSF file:

## Run script to copy buildver_pkg.vhd file and update revision register
set_global_assignment -name PRE_FLOW_SCRIPT_FILE "quartus_sh:update_build_stamp.tcl"

Now when the design is built using the build flows in the GUI, this script will be executed first and will update the revision register with the current system time.

This does not allow for subsequent builds from the makefile-generated-subdirectory area without modifying the revision register, though. I suppose we could implement an additional, less automated major revision register that is not automagically updated by a script in the GUI... but now we're circling back.

Cygwin and File Permissions

I was running into a problem while setting up command line scripting of the Altera tools. Some files generated by SOPC Builder would have permissions of 000 in Cygwin so quartus_map would fail because it couldn't open those files in read mode. 

Apparently Cygwin defaults to using POSIX permissions. Some files generated by Windows apps set ACLs that are then interpreted incorrectly by Cygwin. 

The only work-around I have found is to change the way Cygwin mounts to the filesystem to ignore ACLs. I modified the /etc/fstab in the Cygwin root to uncomment the following line and add 'noacl': 

 none /cygdrive cygdrive binary,noacl,posix=0,user 0 0 

 I had to close down all of my Cygwin processes and relaunch them to fix the problem.