Command Line Basics

One of my favorite features of computers is making them do the work for me.  I tend to think of this as automation.  Command line and shell scripts are a great way to automate tasks.  While Apple has "Automator" and Windows have DOS Batch and PowerShell, Unix shell is my favorite due to its power, availability, and even its philosophy.

Unix shell commands are famous for a philosophy of "do one thing, do it well", and permit chaining of commands to accomplish more complex tasks.  Breaking complex tasks into discrete steps is a good practice for programming, also.

Before tackling full-blown scripts, it is easier to start by "playing" with the individual commands.  Having a good terminal program can greatly aid this.  Terminals are programs that provide access to the computer's shell.  "Console" is the name for a special connection to the first shell of a user's login, but is  otherwise like other shell sessions.

A good terminal application will provide buffers for scrolling through command & output history, make it easy to copy & paste text.  For example, the DOS Command Prompt is pretty painful for copy/paste (menu "Edit", then copy or paste) while in some programs, a double-click can select entire path & filenames while a triple-click can select the entire line.  Macs ship with "Terminal.app", but I prefer to use "iTerm2".  When working on Windows machines, I prefer to install "Console2".  You can try out alternatives and find your favorite.

Unix shells are actually an entire set of applications that interpret commands and built-ins (basically look like commands, but are "built in" to the shell.  There are technically many different shells that have different rules for variables, types, globs - but that is outside the scope of this post.  The shell will "prompt" the user for a command.  The default prompt is "$", while a prompt of ">>" indicates the shell expects more text before it can process the command(s).  However, both are configurable.

Filenames can be absolute (the entire path from that computers' perspective) or relative to the current directory.  In order to enhance security, most Unix systems will set the path to NOT include the current directory so you don't accidentally run a copy of a program that happens to exist in the current directory instead of its "normal" location.  In order to tell the shell you really mean to run the copy in your current directory, prepend './' to the program/script name (meaning the path is HERE).

If you want to pass in a filename to the command, you can pass in the exact filename, a filename pattern (such as *.pdf for any PDF in the current directory) , or even a "glob" (fancier pattern, such as **/*.pdf for any PDF in the current or any child sub-directory at any depth below).  Technically, all 3 are globs. And the fancier glob examples depend on your shell and configuration.

Without further ado, here is a manageable list of unix shell commands that all users should be familiar with.  At first, they may look cryptic, but that is only because they are abbreviations to make it faster to type them.  Once you start dabbling with their options, they may not be so short anymore...

Listing Files

Command	Example	Description
ls	ls –lrtF [dir or file(s)]	LiSt.  List file(s) in a directory, with flags for order and extra information
cd	cd .. cd - cd dir	ChangeDirectory.  “..” is up 1 dir, “-“ is previous dir NOTE: Unix uses “/” instead of Window’s “\”
mkdir	mkdir newDir	MaKeDIRectory.
cp	cp currentFile newFile	CoPy.  If destination exists, it will get over-written.
mv	mv currentFile newFile	MoVe.  (also works for renaming files).  If destination exists, it gets over-written.
rm	rm fileToDelete	ReMove.  Deletes a file or set of files.

Viewing Text File Contents

Command	Example	Description
cat	cat file1 file2 cat file1 file2 > file3	concatenate.  Can dump multiple files to the screen (or get re-directed to another file).
more	more longFile	Displays file 1 page at a time.  Page is dynamic to PuTTY screen.  Advance using <space>, quit by typing “q”.  Also see "less"
view	view textFile	Read only vi (text editor).  Has advanced searching, scrolling, etc.
grep	grep phrase fileToSearch	Search inside files.  Can be literal strings, or advanced regexp (regular expressions – a huge topic by itself).
head	head -10 file1	Display first several lines of a file
tail	tail -10 file1	Display last several lines of a file

Documentation

Command	Example	Description
man	man command man -k keyword	MANual.  If you know the command, you can display man pages for it.  If you don't know the command name, use "-k" to search for related keywords to find the command

command help

<cmd> -h <cmd> --help

By convention, commands with the option "-h" or "--help" should provide their own documentation. This applies to scripts you develop, too.

Execution/Job Control

Command	Example	Description
ps	ps ps -efx	ProccesseS.  List running processes and their IDs.  Each platform often has different flags/options.
nohup	nohup command nohup ./localScript.sh	Prevent interruption.  Prepend to other commands so they can’t be killed or accidentally stopped (ie lost terminal connection)
kill	kill -9 <process id>	Kill a running process by ID (often called "pid")
&	./runLoad.sh &	Background a command.  Returns you to the prompt for other commands and lets first command run in “background”

Introductions - passing my context in

"Standing on the shoulders of giants" is a well-known phrase people use to indicate they have gotten to their current location thanks to the hard work of many others before them.  Open Source Software (OSS) codifies that concept, encouraging us all to share, and provide a better starting place for those behind us.

I have long had a habit of lurking on the edges, processing what I could pick up from others brave enough to speak.  I had the attitude "Better to remain silent and be thought a fool than to speak out and remove all doubt."  However, lately I have started to agree with a line of thought that it is ok to share ideas that may even be wrong - as long as I am willing to update based on conversations around the topic.  So, that gave me the impetus to start this blog.

I thought it would help to identify where I have come from, and what lessons I have picked up along the way.  Hopefully, I can expand on some of the lessons in future posts.

From school:

1. Actively seek learning opportunities
2. Value of internships pursuing your career before you get there
3. Lessons evolve over time - if you make time to ponder.

Factory Automation & Motion Control:

1. Rounding Errors caused by digital sampling
2. The process of Troubleshooting
3. Engineering Estimates for sanity checks

Software Quality Assurance & Testing:

1. Equivalence Classes
2. Test Plans
3. Metrics will get gamed
4. Processes - the wrong way (looking at you, Mr Waterfall).  Setting the stage for agile

 Shells and Scripting:

1. Let the computer do the work for you.  Beginnings of DRY
2. Repeatability is even more valuable than speed
3. Favorite shell commands

Databases: Queries, Data Warehouses and ETL:

1. Set Math: union, intersects, etc
2. In databases, null <> null
3. Sub-Queries for readability & modularity
4. Religion is overrated (Oracle vs SQL Server)
5. Performance Tuning such as inserts cheaper than updates.
6. Profiling Data - helps identify which inputs exist, what/how much should exist in output
7. Extract Transform Load.  Consistent approach helps decompose any conversion of data.
8. Hammers.  People have their comfort zone.  It doesn't mean those are appropriate solutions.

Web Development:

1. HTML5 & Avoiding the lowest common denominator
2. Find smart resources, including people smarter than you
3. Find existing solutions.