Revision as of 13:27, 9 July 2015

Graphical user interfaces (GUIs) are helpful for many tasks, but they box you in in the tasks the designer designed the GUI for. This is true to a certain extent for the command-line too, as it relies on the commands available. Still, some commands are so basic (close to the kernel), and come with many flags and options, or can be built on easily and be combined with other commands in shell scripts, that knowing the command-line and shell scripting is well worth the effort.

Since bodies and machines are often seen in opposition, I suggest that they are better perceived complementary in nature rather than antagonistic. For people who have never worked with command line computing on a standard *nix machine, – especially for people who are already conditioned to point and click methods cultivated by GUIs such as Windows OS or Mac OS – this involves sensitising procedures, (i.e. like one may endure with any new instrumental skill acquisition) for the operation of code as a series of interrelated programs. I will discuss how using the command line interface may be seen to possibly co-constitute one another in everyday life, operating as fields of embodied reflection. ^[1]

Farce of the Pinguins: A mockumentary that illuminates penguin survival and mating rituals, as well as one bird's search for love while on a 70-mile trek with his hedonistic buddies http://www.imdb.com/title/tt0488539/

Command Line Culture (CLC)

Some people use a Command Line Interface (CLI) extensively, and like it more than a GUI. After a ten-step program, they will admit something like, "I am a command line junkie, I like it far better than pointing and clicking. I have become addicted to the bash command, and the basic linux utilities. I find myself installing the basic GNU tools on any system I use. Heck I even installed cygnus-win on my windows gaming box. Mmmm... Command completion... Tasty!"

A only somewhat more sane version of that seems to be running a GUI and a command line at the same time and switching between the two depending on what needs doing. Usually things can be done faster with the command line, but there are situations, such as doing something with multiple directories, when a GUI is more efficient.

$ cd /insanely/long/directory/path/and/you/thought/you/were/there/yet/but/no/muhhahahaaa/aaaaah

Typing that tends to waste time, even when using that yummy command completion. When doing that same thing regularly with the GUI, that may get annoying too and scripting ensues. Goodbye ten step plan. :D Well, except for geany.

Getting started

• Case sensitivity is very important and a common source of problems for people new to Linux. Other systems such as M$ windows are case insensitive when it comes to referring to files. Linux is not like this. You can have two or more files and directories with the same name but letters of different case.
• A lof of commands in linux are named as an abbreviation of a word or words describing them. This makes it easier to remember them.

Man

Bash shells come with a very useful utility called man, short for manual files or manual pages. It gives a standardised format for documenting the purpose and usage of most of the utilities, libraries, and system calls https://www.kernel.org/doc/man-pages/. For documentation other than man pages, see the Linux Documentation Project site http://www.tldp.org/.

The manual pages are a set of pages that explain every command available on your system including what they do, the specifics of how you run them and what command line arguments they accept. They are fairly consistent in their structure so you can easily get the hang of it. Start up a console or terminal and invoke the manual pages with the following command:

$ man [command]

For example:

$ man grep

Everything is a file

Everything in linux can be viewed as a file:

• regular files are documents, images, archives, recordings, directories (just a file containing names of other files) …
• (character and block) device files give you access to hardware components
• named pipes and sockets give access points for processes to communicate with each other
• (hard and soft) links make a file accessible from different locations

Navigation

With pwd (present working directory) you can see your location in the file structure.

$ pwd 
/home/user
$

With ls (list) you can see what is in a location:

$ ls [options] [location]

For example:

$ ls -l /home/user
total 20
drwxr-xr-x 2 user user 4096 Jun 17 14:39 Desktop
drwxr-xr-x 2 user user 4096 Jul  2 00:45 Documents
drwxr-xr-x 4 user user 4096 Jul  2 00:46 Pictures

The result lines explained

• The first character on a result line indicates whether it is a normal file - or a directory d. In the above example all are directories.
• The next 9 characters are permissions for the file or directory. More on that in file permissions below.
• A character representing the number of blocks.
• The field following that is the owner of the file or directory (user in this case).
• The group the file or directory belongs to (user)
• File size
• File modification time
• Name of the file or directory

For more explanation on and examples of using ls do:

$ man ls

When referring to either a file or directory on the command line, like with /home/user in the ls example, we are referring to a path, a description of a route to get to a particular file or directory on the system. The linux file system is a hierarchical with at the very top of the structure a directory called the root directory denoted by a single slash /. It has subdirectories and the subdirectories have subdirectories and so on. Files may reside in any of these directories.

Paths can be absolute or relative:

• Absolute paths specify a location (file or directory) in relation to the root directory and begin with a /
• Relative paths specify a location (file or directory) in relation to where you currently are in the system and do not begin with a /

More building blocks:

• ~ (tilde), a shortcut for your home directory. For example /home/user/Pictures and ~/Pictures both refer to the Pictures folder in the home directory of user.
• . (dot), a reference to your current directory. For example, ./Pictures refers to the same directory again.
• .. (dotdot), a reference to the parent directory. You can use this several times in a path to keep going up the hierarchy. If you are in the location the path /home/user refers to, you could run the command ls ../../ and this would return a listing of the root directory.

In order to move around in the system you can use a command called cd (change directory):

$ cd [location]

Typing out these paths can become tedious. Not to mention my typos. Yessss. Tab Completion. When you start typing a path and hit the Tab key on your keyboard at any time you will invoke an auto complete action. If nothing happens then that means there are several possibilities. If you hit Tab again it will show you those possibilities. You can continue typing and hit Tab again and it will again try to auto complete for you.

File manipulation

From the command line, there are many ways to create, find and list different types of files.

In systems such as M$ Windows the extension is important and the system uses it to determine what type of file it is. In linux the system ignores extensions and looks inside the file to determine what type of file it is. So sometimes it can be hard to know for certain what type of file a particular file is. You can determine the type of a file with the file command:

$ file privatelyinvestigating.wordpress.2015-05-02.xml 
privatelyinvestigating.wordpress.2015-05-02.xml: XML document text

With cp (copy) you can copy files and directories:

$ cp [options] [filename] [filename]

For example:

$ cp -u *.png /home/user/Pictures/

Will copy all files in the current directory with extension .png to the Pictures directory in the home directory of user.

With mv (move) you can move or rename files and directories. To rename a file, use like this:

$ mv [filename1] [filename2]

To move a file, use like this:

$ mv [filename1] [directory]

To move files, use like this:

$ mv [filename1] [filename2] [directory]

With rm (remove) you can remove files and directories. Linux does not have an undelete command. Once you delete something with rm, it's gone. You can inflict horrifying damage on your system with rm if you are not careful, particularly with wildcards such as *.

To remove a file:

$ rm [filename]

To remove directories:

$ rm -r [filename]

And with mkdir (make directory) you can create directories:

$ mkdir [directory]

File permissions

Unix-like operating systems differ from other computing systems in that they are not only multitasking but also multi-user. The multi-user capability of Unix-like systems is a feature that is deeply ingrained into the design of the *nix operating system. In the environment in which Unix was created, this makes perfect sense, and now, with the internet, this makes perfect sense again. In the beginning, computers were large, expensive, and centralised, access was by terminals and The Computer would support many users at the same time, as does the internet. A method had to be devised to protect users from each other.

In linux, each file and directory is assigned access rights for the owner of the file, the members of a group of related users, and everybody else. Rights can be assigned to read a file, to write a file, and to execute a file (run the file as a program). There are two ways to specify the permissions.

For the first, see the permission settings for a file or directory, use the ls -l command (see above in Navigation). Taking one line of the results:

drwxr-xr-x 4 user user 4096 Jul  2 00:46 Pictures

The ls -l output line starts with a d indicating it is a directory, and the next nine characters are for permissions. These are three groups of three characters each. The first set of three characters rwx is for owner, the owner of the file. Owner has read r, write w and execute x permissions on that directory. The second set of characters is for group. Users in the group have r-x permissions and can only read and execute the file. Other (the rest of the world) have those permissions too in this case.

The conversion to the other permissions representation goes like this:

1) Convert the three sets rwx r-x r-x to three groups of binary code using 1's for "turned on" indicated by r, w and x and 0's as "turned off" indicated by a -, like so:

rwx = (111)₂

r-x = (101)₂

r-- = (100)₂

--x = (001)₂

The example then looks like 111 101 101₂

2) Convert binary code to octal code. If not familiar with number conversions, a decent tutorial can be found in http://www.cstutoringcenter.com/tutorials/general/convert.php

(111)₂ = 2⁰ + 2¹ + 2² = 1 + 2 + 2*2 = 7₈

(101)₂ = 2⁰ + 2² = 1 + 2*2 = 5₈

(100)₂ = 2² = 2*2 = 4₈

(001)₂ = 2⁰ = 1 ₈

The example then looks like 7 5 5₈. So the permissions for this Pictures directory in octal notation are 755 ₈ and the base, 8, is often not mentioned.

Handy file permissions mental shortcuts:

• 777 (rwxrwxrwx) No restrictions on permissions. Anybody may do anything. Generally not a desirable setting.
• 755 (rwxr-xr-x) The file's owner may read, write, and execute the file. All others may read and execute the file. This setting is common for programs that are used by all users.
• 700 (rwx------) The file's owner may read, write, and execute the file. Nobody else has any rights. This setting is useful for programs that only the owner may use and must be kept private from others.
• 666 (rw-rw-rw-) All users may read and write the file. But not execute.
• 644 (rw-r--r--) The owner may read and write a file, while all others may only read the file. A common setting for data files that everybody may read, but only the owner may change.
• 600 (rw-------) The owner may read and write a file. All others have no rights. A common setting for data files that the owner wants to keep private.

Handy directory permissions mental shortcuts:

• 777 (rwxrwxrwx) No restrictions on permissions. Anybody may list files, create new files in the directory and delete files in the directory. Generally not a good setting.
• 755 (rwxr-xr-x) The directory owner has full access. All others may list the directory, but cannot create files nor delete them. This setting is common for directories that you wish to share with other users.
• 700 (rwx------) The directory owner has full access. Nobody else has any rights. This setting is useful for directories that only the owner may use and must be kept private from others.

With chmod you can modify access rights to a file:

$ chmod [permissions] [filename]

With su (super user) or sudo (as superuser do) you can temporarily become the superuser. Doing su on debian you will be asked for your root password.

$ su 

A new shell owned by root is started, indicated by a # instead of a $ as prompt. You can kill that shell and return to your previous user shell with exit:

# exit

In mint and ubuntu default you do not have su but you can use sudo. You will be asked for your user password.

$ sudo

In order to change the owner of a file, you must be the superuser. With chown (change ownership) you can then change file ownership:

# chown [username] [filename] 

For changing the group ownership of a file you do not need superuser, but you do need to be owner of the file. With chgrp (change group) you can change a file's group ownership:

$ chgrp [groupname] [filename]

Regular expressions

Regular expressions are strings that describe a collection of strings using a for that purpose created language. That probably reads like garble, but a few examples can help. Regular expressions are useful for expansion, static code source analysis, reverse engineering, malware fingerprinting, vulnerability assessment, and exploit development.

Pathname expansion

Tilde expansion

When a tilde is used at the beginning of a word, it expands into the name of the home directory of the named user, or if no user is named, the home directory of the current user:

$ echo ~ 
/home/user 

If user "foo" has an account, then:

$ echo ~foo 
/home/foo

Arithmetic expansion

The shell allows arithmetic to be performed by expansion making using the shell prompt as a calculator easy. Arithmetic expansion only supports integers (whole numbers, no decimals), but can perform quite a number of different operations::

$ echo $((2 + 2)) 
4

It allows for nesting of expressions (5**2 means 5²):

$ echo $(($((5**2)) * 3)) 
75 

Single parentheses may be used to group multiple subexpressions:

$ echo $(((5**2) * 3)) 
75 

Here is an example using the division and remainder operators (integer division):

$ echo Five divided by two equals $((5/2)) 
Five divided by two equals 2 
$ echo with $((5%2)) left over. 
with 1 left over. 

Brace expansion

Perhaps the strangest expansion is called brace expansion. The brace expansion is present in two basic forms, string lists and ranges. It can be switched on and off under runtime by using the set builtin and the option -B and +B or the long option braceexpand. If brace expansion is enabled, the stringlist in SHELLOPTIONS contains braceexpand.

You can create multiple text strings from a pattern containing braces:

$ echo last{mce,boot,xorg}.log
lastmce.log lastboot.log lastxorg.log

Patterns to be brace expanded may contain a leading portion called a preamble and a trailing portion called a postscript. The brace expression itself may contain either a comma-separated list of strings, or a range of integers or single characters. The pattern may not contain embedded whitespace.

Using a range of integers in reverse order:

$ echo Number_{5..1} 
Number_5 Number_4 Number_3 Number_2 Number_1 

Brace expansions may also be nested:

$ echo a{A{1,2},B{3,4}}b

aA1b aA2b aB3b aB4b

The most common application of brace expansion is to easily make lists of files or directories to be created:

$ mkdir {2011..2013}-0{1..9} {2011..2013}-{10..12} 
$ ls 
2011-01 2011-07 2012-01 2012-07 2013-01 2013-07
2011-02 2011-08 2012-02 2012-08 2013-02 2013-08
2011-03 2011-09 2012-03 2012-09 2013-03 2013-09
2011-04 2011-10 2012-04 2012-10 2013-04 2013-10
2011-05 2011-11 2012-05 2012-11 2013-05 2013-11
2011-06 2011-12 2012-06 2012-12 2013-06 2013-12

Input/Output redirection

I/O redirection is one of the easiest things to master. It allows for combining different utilities effectively. For example, you may want to search through the output from nmap or tcpdump or a key-logger by feeding its output to another file or program for further analysis.

Search for network traffic

Grep can be used with tcpdump to search for specific network traffic:

$ sudo tcpdump -n -A | grep -e 'POST'
[sudo] password for user: 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
E...=.@.@......e@.H..'.P(.o%~...P.9.PN..POST /blog/wp-admin/admin-ajax.php HTTP/1.1
E...c_@.@..=...e@.H..*.PfC<....wP.9.PN..POST /blog/wp-admin/admin-ajax.php HTTP/1.1
E.....@.@......e@.H...."g;.(.-,WP.9.Nj..POST /login/?login_only=1 HTTP/1.1

Sniffing passwords using egrep

$ tcpdump port http or port ftp or port smtp or port imap or port pop3 -l -A | egrep -i 'pass=|pwd=|log=|login=|user=|username=|pw=|passw=|passwd=|password=
|pass:|user:|username:|password:|login:|pass|user' --color=auto --line-buffered -B20

Process management (job control)

When an executable program starts up, it runs as a process under management of the process table. The ps and top command can be used to look at running processes; nice and renice for raising and lowering priority of a process; processes can be moved to run in the background with bg or to the foreground with fg; kill and killall can be used to send signals to a process; stop and (re)start a process; reread configuration files; and cron can run commands at a scheduled time.

The command pgrep looks through the currently running processes and lists the process IDs matching the selection criteria to stdout. All criteria have to match.

For listing all processes named ssh AND owned by root:

$ pgrep -u root ssh

Listing processes owned by root OR deamon:

$ pgrep -u root,daemon

Changing process priority

Adding jobs to cron

Shell scripting

Shell scripts are good for automating repetitive shell tasks. Bash and other shells include the “usual” constructs found in programming languages, such as for loops, tests, if and case statements, but there is only one type of variable: strings.

Shells

Unix has variety of Shells. Bourne shell (sh), Bourne again shell (bash), C shell (csh), Korn shell (ksh), Tenex C shell (tcsh). Use the which or whereis unix commands to find out where a specific shell is located:

# which bash
/bin/bash

or:

# whereis bash
bash: /bin/bash /etc/bash.bashrc /usr/share/man/man1/bash.1.gz

Hello World bash script

Print system related information

Backup and restore using brace expansion

When doing:

$ echo {a,b}$PATH

the brace expansion does not expand the variable - this is done in a later step. Brace expansion just makes it being:

echo a$PATH b$PATH

Another common pitfall is to assume that a range like {1..200} can be expressed with variables using {$a..$b}. It simply is not possible, because it's the very first step in doing expansions. A possible way to achieve this, is using the eval command, which basically evaluates a commandline twice:

eval echo {$a..$b}

For instance, when embedded inside a for loop in a shell script:

for i in $(eval echo {$a..$b})

Backup script using brace expansion:

Restore script using brace expansion:

Network connections

Connecting to a network from a linux box is easy, and on occasion not. If a network interface does not come up or requires manual setup, there are many commands for configuring interfaces, checking network connections and setting up special routing. Once connection is up there are more commands for getting information about networks your machine is connected to.

Monitoring network connections

Configuring network interfaces

Reconnaissance

Querying DNS servers

The whois system is used by system administrators to obtain contact information for IP address assignments or domain name administrators. Dig is a networking tool that can query DNS servers for information. It can be very helpful for diagnosing problems with domain pointing and is a good way to verify that your server configuration is working. An alternative to dig is a command called host. This command functions in a very similar way to dig, with many of the same options. And if dig and whois do not provide you with enough information, tools like dnsmap and dnsenum can be handy.

Enumerating targets

Enumerating targets on your local network can be done with nmap, arping, hping and fping. The last three allow for constructing arbitrary packets for almost any networking protocol, for analysis of replies.

Arp sweeps

If you regularly do arp sweeps of your network, you can use arp-scan or develop your own from this basic script:

#!/bin/bash
PREFIX=$1
INTERFACE=$2
for SUBNET in {1..255}
do
   for HOST in {1..255}
   do
      echo "[*] IP : "$PREFIX"."$SUBNET"."$HOST
      arping –c 3 –i $INTERFACE $PREFIX"."$SUBNET"."$HOST 2> /dev/null
   done
done

If named arpsweep (or arpsweep.sh) call with (for example):

$ arpsweep 192.168 eth0

Reverse engineering

Learn about reverse engineering and backdooring hosts, discover memory corruption, code injection, and general data- or file-handling flaws that may be used to instantiate arbitrary code execution vulnerabilities.

Metasploit

First some preps that make life a little easier. Metasploit can be used in the environment of the bash shell.

Disassembly

Disassembly is the process of reversing the effect of code compilation as much as possible. And does not make sense at all if you know nothing about the parts of your processor that are made visible to machine instructions. Minimally you need to know about its registers (which can be bit-vector/integer, floating point, machine address), how Arithmetic Logic Units work, how clocking circuits works and why some instructions take more than one clock, how first and second level caches work, how Memory Management Units and Direct Memory Access work, etc.

Network exploitation and monitoring

Warning: Do not execute these on a network or system that you do not own. Execute only on your own network or system for learning purposes. Do not execute these on any production network or system.

Spoofing

Questioning servers

Brute-forcing authentication

Traffic filtering

Testing SSL implementation

Related

• Installing linux
• Linux security

References

1. ↑ Linux for Theatre Makers: Embodiment and *nix modus operandi http://networkcultures.org/blog/2007/04/23/linux-for-theatre-makers-embodiment-and-nix-modus-operandi/