Sort and tsort file sorting in bash

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML xmlns:fo = "http://www.w3.org/1999/XSL/Format"><HEAD><TITLE>Tip: Sorting files with sort and tsort</TITLE> <META http-equiv=Content-Type content="text/html; charset=UTF-8"> <META http-equiv=PICS-Label content='(PICS-1.1 "http://www.icra.org/ratingsv02.html" l gen true r (cz 1 lz 1 nz 1 oz 1 vz 1) "http://www.rsac.org/ratingsv01.html" l gen true r (n 0 s 0 v 0 l 0) "http://www.classify.org/safesurf/" l gen true r (SS~~000 1))'> <META http-equiv=Expires content=0> <META content="text utilities textutils GNU command commands command-line tool tools ptx tr uniq fold expand fmt pr wc checksum checksums cksum sum od comm diff headtail cut split less more cat sort tsort nl jacek artymiak IBM developerWorks tips hints" name=keywords> <META content="Save time and headaches by using sort and tsort -- instead of resorting to more complex solutions utilizing Perl or Awk." name=ABSTRACT> <META content="Save time and headaches by using sort and tsort -- instead of resorting to more complex solutions utilizing Perl or Awk." name=DESCRIPTION> <META content="Tip: Sorting files with sort and tsort" name=TITLE> <META content=https://www-136.ibm.com/developerworks/secure/feedback.jsp?domain=developerworks name=OWNER> <META scheme=iso8601 content=2003-3-6 name=DC.Date> <META scheme=rfc1766 content=en-us name=DC.Language> <META content=zz name=IBM.Country> <META content=Public name=SECURITY> <META content="Copyright (c) 2003 by IBM Corporation" name=DC.Rights> <META content=index,follow name=ROBOTS> <SCRIPT language=JavaScript src="Sorting files with sort and tsort_files/style.js" type=text/javascript></SCRIPT>

<SCRIPT language=JavaScript type=text/javascript>var title = "Tip: Sorting files with sort and tsort"; </SCRIPT>

<SCRIPT language=JavaScript type=text/javascript>var forumURL = ""; </SCRIPT>

<SCRIPT language=JavaScript type=text/javascript>var contentAreaList = "linux"; </SCRIPT>

<SCRIPT language=JavaScript type=text/javascript>var emailAbstract = "Save time and headaches by using sort and tsort -- instead of resorting to more complex solutions utilizing Perl or Awk."; </SCRIPT>

<SCRIPT language=JavaScript type=text/javascript>var demoURL = ""; </SCRIPT>

<A

     href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=http://www.ibm.com/&origin=dwheader"><IMG 
     height=47 alt=IBM 
     src="Sorting files with sort and tsort_files/ibm-logo.gif" width=160

border=0></A>

<A

     href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex4.html#main"><IMG 
     height=1 alt="Skip to main content" 
     src="Sorting files with sort and tsort_files/c.gif" width=70

border=0></A>

<IMG height=4 alt="" src="Sorting files with sort and tsort_files/c.gif" width=390 border=0>
Search for:	<INPUT class=dwsearch maxLength=100 size=18 name=query height="15">	within	<SELECT class=dwsearch name=searchScope> <OPTION value=dW selected>All of dW</OPTION> <OPTION value=dW>-----------------</OPTION> <OPTION value=db2> DB2</OPTION> <OPTION value=eserver> eServer</OPTION> <OPTION value=lotus> Lotus</OPTION> <OPTION value=rdd> Rational</OPTION> <OPTION value=tivoli> Tivoli</OPTION> <OPTION value=WSDD> WebSphere</OPTION> <OPTION value=dW>-----------------</OPTION> <OPTION value=gridZ> Grid computing</OPTION> <OPTION value=javaZ> Java technology</OPTION> <OPTION value=linuxZ> Linux</OPTION> <OPTION value=opensrcZ> Open source</OPTION> <OPTION value=webarchZ> Web arch.</OPTION> <OPTION value=webservZ> Web services</OPTION> <OPTION value=wirelessZ> Wireless</OPTION> <OPTION value=xmlZ> XML</OPTION> <OPTION value=dW>-----------------</OPTION> <OPTION value=forums> dW forums</OPTION> <OPTION value=dW>-----------------</OPTION> <OPTION value=toolbox> Toolbox</OPTION> <OPTION value=toolbox> subscription</OPTION> <OPTION value=dW>-----------------</OPTION> <OPTION value=aW>alphaWorks</OPTION> <OPTION value=dW>-----------------</OPTION> <OPTION value=all>All of IBM</OPTION></SELECT>	<IMG height=1 alt="" src="Sorting files with sort and tsort_files/c.gif" width=5 border=0>	<INPUT type=image height=23 alt="Search button" width=64 src="Sorting files with sort and tsort_files/search.gif" value=Search border=0 name=Search>	<IMG height=1 alt="" src="Sorting files with sort and tsort_files/c.gif" width=10 border=0>
			<A style="COLOR: #ffffff" href="http://www-106.ibm.com/developerworks/search/help-dw.html" target=_blank>Search help</A><IMG height=1 alt="" src="Sorting files with sort and tsort_files/c.gif" width=1 border=0>
<IMG height=4 alt="" src="Sorting files with sort and tsort_files/c.gif" width=390 border=0>

<A

     class=mainlink 
     href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=http://www.ibm.com/&origin=dwheader">IBM 
     home</A>  |  <A 
     class=mainlink 
     href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=http://www.ibm.com/products/&origin=dwheader">Products 
     & services</A>  |  <A 
     class=mainlink 
     href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=http://www.ibm.com/support/">Support 
     & downloads</A>   |  <A 
     class=mainlink 
     href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=http://www.ibm.com/account/&origin=dwheader">My

account</A>

<IMG height=4 alt=""

           src="Sorting files with sort and tsort_files/c.gif" width=2 
           border=0>
<A class=dwbctl 
           href="http://www-106.ibm.com/developerworks/">developerWorks</A> 
           > <A class=dwbctl

href="http://www-106.ibm.com/developerworks/linux/">Linux</A>

<A

           href="http://www-106.ibm.com/developerworks/"><IMG height=24 
           alt=developerWorks 
           src="Sorting files with sort and tsort_files/dwlogo-small.gif"

width=136 border=0></A>

<IMG height=1 alt=""

           src="Sorting files with sort and tsort_files/c.gif" width=5

border=0>

<A name=main></A><A

name=top></A>

<IMG height=15 alt="" src="Sorting files with sort and tsort_files/c.gif" width=5 border=0>
<IMG height=1 alt="" src="Sorting files with sort and tsort_files/c.gif" width=2 border=0>	Tip: Sorting files with sort and tsort	<IMG height=1 alt="" src="Sorting files with sort and tsort_files/c.gif" width=8 border=0>	<IMG height=1 alt="" src="Sorting files with sort and tsort_files/c.gif" width=180 border=0> <NOBR><A href="javascript:void%20newWindow()"><IMG height=26 alt="e-mail it!" src="Sorting files with sort and tsort_files/icon-email.gif" width=46 border=0></A></NOBR>	<IMG height=1 alt="" src="Sorting files with sort and tsort_files/c.gif" width=6 border=0>
<IMG height=1 alt="" src="Sorting files with sort and tsort_files/c.gif" width=100 border=0>
<IMG height=8 alt="" src="Sorting files with sort and tsort_files/c.gif" width=100 border=0>

<IMG height=1 alt=""

                 src="Sorting files with sort and tsort_files/c.gif"

width=160>

Contents:

<IMG height=1 alt=""

                 src="Sorting files with sort and tsort_files/c.gif"

width=160>

<A href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex4.html#resources">Resources</A>

<IMG height=5 alt=""

                       src="Sorting files with sort and tsort_files/c.gif"

width=160>

<A

                       href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex4.html#author1">About

the author</A>

<IMG height=5 alt=""

                       src="Sorting files with sort and tsort_files/c.gif"

width=160>

<A

                       href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex4.html#rating">Rate

this article</A>

<IMG height=10 alt=""

                       src="Sorting files with sort and tsort_files/c.gif"

width=160>

<IMG height=1 alt=""

                 src="Sorting files with sort and tsort_files/c.gif"

width=160>

Related content:

<IMG height=1 alt=""

                 src="Sorting files with sort and tsort_files/c.gif"

width=160>

<A

                       href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex1.html">Get

to know your textutils</A>

<IMG height=5 alt=""

                       src="Sorting files with sort and tsort_files/c.gif"

width=160>

<A

                       href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex2.html">Concatenating

files with cat</A>

<IMG height=5 alt=""

                       src="Sorting files with sort and tsort_files/c.gif"

width=160>

<A

                       href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex3.html">Reading

text streams in chunks with head and tail</A>

<IMG height=5 alt=""

                       src="Sorting files with sort and tsort_files/c.gif"

width=160>

<IMG height=5 alt=""

                       src="Sorting files with sort and tsort_files/c.gif"

width=160>

<IMG height=1 alt=""

                 src="Sorting files with sort and tsort_files/c.gif"

width=160>

Subscriptions:

<IMG height=1 alt=""

                 src="Sorting files with sort and tsort_files/c.gif"

width=160>

<A

                       href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=www-106.ibm.com/developerworks/newsletter/&origin=dw-article">dW

newsletters</A>

<IMG height=5 alt=""

                       src="Sorting files with sort and tsort_files/c.gif"

width=160>

<A

                       href="http://www-106.ibm.com/developerworks/toolbox/">dW

Subscription
(CDs and downloads)</A>

<IMG height=5 alt=""

                       src="Sorting files with sort and tsort_files/c.gif"

width=160>

<IMG height=2 alt="" src="Sorting files with sort and tsort_files/c.gif" width=160>
<IMG height=2 alt="" src="Sorting files with sort and tsort_files/c.gif" width=160>

Get to know your textutils
<TBODY> </TBODY>

Level: Introductory

<A href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex4.html#author1"><NAME>Jacek Artymiak</NAME></A> (<A href="mailto:jacek@artymiak.com?cc=&subject=Tip: Sorting files with sort and tsort">mailto:jacek@artymiak.com?cc=&subject=Tip: Sorting files with sort and tsort</A>)
Freelance author and consultant
6 March 2003

<ABSTRACT-EXTENDED>Save time and headaches by using
sort and tsort -- instead of resorting to more complex solutions utilizing Perl or Awk. Jacek Artymiak explains
how.</ABSTRACT-EXTENDED>

Although it is possible to write advanced sorting applications in Perl or Awk, doing so may not always be necessary -- and is often a pain. Most things you'll ever need are equally possible -- and lots easier -- with the sort command, which can sort lines in more than one file, merge files, and even check to see if sorting them is necessary. You can specify sort keys (portions of lines used for comparisons) or not, in which case sort just compares whole lines.

So, if you want to sort your password file, you could just use the following. (Note that you cannot send the output straight to the input file, because it will corrupt the input file. That's why you need to send it to a temporary file and then rename that file to /etc/passwd -- as shown below.)

<A name=code1>Listing 1. Simple sort</A>
<TBODY> </TBODY>

<CODE>
        
$ su - 
# sort /etc/passwd > /etc/passwd-new
# mv /etc/passwd-new /etc/passwd
      </CODE>

<A name=side>More on sort and tsort</A>

Follow along in

           the man page by opening the <A 
           href="http://www.gnu.org/manual/textutils-2.0/html_chapter/textutils_7.html#SEC22" 
           target=new>GNU manual's pages on sort operations</A>, or view these 
           options in your man or info pages in a new terminal window by typing 
           man sort or man tsort at the command line.

Should you want to reverse the order of sorting, use the -r option. You can also suppress printing of identical lines with the -u option.

A very practical feature of sort is its ability to sort using field keys. A field is a string of text separated from other fields with a certain single character. For example, the fields in /etc/passwd are separated with a colon (:). So, if you wanted, you could sort /etc/passwd by the user ID, group ID, comments field, home catalog, or shell. To do this, use the -t option followed by the character used as the separator, and the number of the field that will be used as the sort key followed by the number of the last field where the key will end; for example, sort -t : -k 5,5 /etc/passwd sorts the password file by the comment field, which is the place where full user names like "John Smith" are stored. But sort -t : -k 3,4 /etc/passwd sorts the same file using both the user ID and the group ID. If you omit the second number, sort will assume that the key starts at the given field and continues to the end of each line. Try this yourself, and observe the differences. (When numeric sorting looks wrong, add the -g option).

Also, note that a whitespace transition is the default separator -- so if fields are already separated by blank characters, you may omit the separator and use -t alone. (Notice also that numbering of fields starts with 1.)

For even finer control, you can use keys and offsets. Offsets are separated from keys with a dot, as in -k 1.3,5.7, which means that the sort key should start on the third character of the first field, and end at the seventh character of the fifth field (offsets too are numbered from 1). When would you need this? Well, I use it from time to time for sorting Apache logs; the key and offset notation lets me skip the date fields.

Another option to watch out for is -b, which tells sort to ignore blank characters (spaces, tabs, etc.) and treat the first non-blank character on the line as the start of the sort key. Also, if you use that option, offsets will be counted from the first non-blank character (useful when the field separator is not a blank character and when the fields may contain strings starting with blank characters).

Further modifications of the sorting algorithm are possible with these options: -d (use only letters, digits, and blanks for sort keys), -f (turn off case recognition and treat lowercase and uppercase characters as identical), -i (ignores non-printing ASCII characters), -M (sorts lines using three-letter abbreviations of month names: JAN, FEB, MAR, ...), -n (sorts lines using only digits, -, and commas, or other thousands separator). These options, as well as -b and -r, can be used as part of a key number, in which case they apply to that key only and not globally, like they do when they are used outside key definitions.

As an example of the use of a key number, consider:

sort -t: -k 4g,4 -k 3gr,3 /etc/passwd

This will sort the passwd file by group ID and within groups by userid, backwards.

But that's not all that sort is capable of. It can also resolve ties that happen when the keys you used cannot be used to decide which line is first. To add hints for resolving ties, add another -k option and follow it with the field and (optional) offset, using the same notation as the one you used for defining keys; for example, sort -k 3.4,4.5 -k 7.3,9.4 /etc/passwd sorts lines using keys that begin at the fourth character of the third key and end at the fifth character of the fourth key and use the third character of the seventh field and the fourth character of the ninth field to resolve ties.

The last group of options deals with input, output, and temporary files. For example, the -c option, when used in sort -c < file, checks if the input file has been sorted yet (you can use other options as well), and if it has, reports an error. This is handy for making checks before processing large files that may take a long time to sort. When you use the -u option together with the -c option, it will be interpreted as a request to check that there are no two identical line in the input file.

Also important when you are processing large files is the -T option used to specify an alternative directory for temporary files (they are removed after sort finishes work) instead of the default /tmp.

You can use sort to process more than one file at a time, and there are basically two ways to do it: you can use cat to concatenate them first, as in:

cat file1 file2 file3 | sort > outfile

Or, you could use this command:

sort -m file1 file2 file3 > outfile

There is one condition in the second case: each input file must be sorted before they are all sent to sort -m together. That may look like an unnecessary burden, but in fact it speeds up work and saves precious system resources. Oh, and don't forget the -m option. You can use the -u option here to suppress printing of identical lines.

If you need a more esoteric kind of sort routine, you might want to check out the tsort command, which performs a topological sort on a file. The difference between a topological and standard sort is shown in Listing 2 (you can download happybirthday.txt from <A href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex4.html#resources">Resources</A>).

<CODE>
        
        <SPAN class=boldcode>$ cat happybirthday.txt</SPAN>

Happy Birthday to You!

Happy Birthday to You!

Happy Birthday Dear Tux!

Happy Birthday to You!

<SPAN class=boldcode>$ sort happybirthday.txt</SPAN>

Happy Birthday Dear Tux!

Happy Birthday to You!

Happy Birthday to You!

Happy Birthday to You!

<SPAN class=boldcode>$ tsort happybirthday.txt</SPAN>

Dear

Happy

to

Tux!

Birthday

You!
</CODE>

Of course, that isn't a very useful demonstration of what you'd use tsort for -- just an illustration of how different the output of the two commands is.

tsort is generally used for solving a logic problem in which it's necessary to predict a total order from observed partial orders; for example, (from the tsort info page):

tsort <<EOF a b c d e f b c d e EOF

will produce the output

<CODE>
      a
      b
      c
      d
      e
      f
</CODE>

Questions or comments? I'd love to hear from you -- send mail to <A href="mailto:jacek@artymiak.com">jacek@artymiak.com</A>.

Next time, we'll delve into tr.

<A name=resources>Resources</A>

Download the example file for Listing 2, <A href="ftp://www6.software.ibm.com/software/developer/library/l-tiptex4/happybirthday.txt">happybirthday.txt</A>.
Find even more info on these useful tools in the <A href="http://www.gnu.org/manual/textutils/html_node/textutils.html">GNU text utilities manual</A>. (An <A href="http://web.mit.edu/gnu/doc/html/textutils_toc.html">expanded view of the same TOC</A> lives at MIT, where you can also find <A href="http://web.mit.edu/gnu/doc/html/menu.html">this great list of even more useful GNU tools</A>.)
Windows users can find these tools in the <A href="http://www.cygwin.com/">Cygwin</A> package.
Mac OS X users may want to try <A href="http://fink.sourceforge.net/">Fink</A>, which installs a rich UNIX environment under the sleek new Mac OS X.
Something just not working for you? Try checking the <A href="http://www.gnu.org/software/fileutils/doc/faq/core-utils-faq.html">Frequently asked questions for GNU textutils</A>.
Need more introductory info before delving in to the tools we've covered here? Try starting with <A href="http://www.eco.utexas.edu/Help/Unixhelp/TOP_.html">UNIXhelp for users</A>.
Of course, the classic work in this field is <A href="http://www.oreilly.com/catalog/upt2/">Unix Power Tools</A>, from O'Reilly and Associates (Jerry Peek, Tim O'Reilly, and Mike Loukides: 1997; ISBN 1-56592-260-3).
Lest we forget, the Jargon File has <A href="http://www.tuxedo.org/~esr/jargon/html/entry/bubble-sort.html">an amusing entry on the topic of sorting</A>.
Read Jacek's other tips in this developerWorks series:
- <A href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex1.html">Get to know your textutils</A>
- <A href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex2.html">Concatenating files with cat</A>
- <A href="http://www-106.ibm.com/developerworks/linux/library/l-tiptex3.html">Reading text streams in chunks with head and tail</A>
Find the Linux resource you're looking for in the <A href="http://www-106.ibm.com/developerworks/linux/">developerWorks Linux zone</A>.

<A name=author1></A>About the

           author
Jacek Artymiak works as a freelance consultant, 
           developer, and writer. Since 1991 he's been developing software for 
           many commercial and free variants of UNIX and BSD operating systems 
           (AIX, HP-UX, IRIX, Solaris, Linux, FreeBSD, NetBSD, OpenBSD, and 
           others), as well as MS-DOS, Microsoft Windows, Mac OS, and Mac OS X. 
           Jacek specializes in business and financial application development, 
           Web design, network security, computer graphics, animation, and 
           multimedia. He's a prolific writer on technology subjects and the 
           coauthor of "Install, Configure, and Customize Slackware Linux" 
           (Prima Tech, 2000) and "StarOffice for Linux Bible" (IDG Books, 
           2000). Many of Jacek's software projects can be found at <A 
           href="http://sourceforge.net/users/artymiak/">SourceForge</A>. You 
           can learn more about him at his <A 
           href="http://www.artymiak.com/">personal Web site</A> and contact 
           him at <A

href="mailto:jacek@artymiak.com">jacek@artymiak.com</A>.

<IMG height=10 alt=""

     src="Sorting files with sort and tsort_files/c.gif" width=100

border=0>

<A

           href="javascript:void%20newWindow()"><IMG height=26 alt="e-mail it!" 
           src="Sorting files with sort and tsort_files/icon-email.gif"

width=46 border=0></A>

<IMG height=1 alt=""

           src="Sorting files with sort and tsort_files/c.gif" width=5

border=0>

<IMG height=1 alt=""

           src="Sorting files with sort and tsort_files/c.gif" width=100

border=0>

<IMG height=8 alt=""

           src="Sorting files with sort and tsort_files/c.gif" width=100

border=0>

           <FORM 
           action=http://www.alphaworks.ibm.com/developerworks/ratings.nsf/RateArticle?CreateDocument 
           method=post><INPUT type=hidden 
           value="Tip: Sorting files with sort and tsort" 
           name=ArticleTitle><INPUT type=hidden value=Linux name=Zone><INPUT 
           type=hidden 
           value=http://www-106.ibm.com/developerworks/thankyou/feedback-thankyou.html 
           name=RedirectURL><A name=rating>What do you think of this 
           document?</A>

<IMG height=8 alt="" src="Sorting files with sort and tsort_files/c.gif" width=100 border=0>
<INPUT type=radio value=5 name=Rating>Killer! (5)	<INPUT type=radio value=4 name=Rating>Good stuff (4)	<INPUT type=radio value=3 name=Rating>So-so; not bad (3)	<INPUT type=radio value=2 name=Rating>Needs work (2)	<INPUT type=radio value=1 name=Rating>Lame! (1)

Comments?
<TEXTAREA name=Comments rows=5 wrap=virtual cols=60></TEXTAREA>

<INPUT type=submit value="Submit feedback"></FORM>

<IMG height=8 alt=""

           src="Sorting files with sort and tsort_files/c.gif" width=100

border=0>

<IMG height=4 alt=""

                 src="Sorting files with sort and tsort_files/c.gif" width=2 
                 border=0>
<A class=dwbctl 
                 href="http://www-106.ibm.com/developerworks/">developerWorks</A> 
                 > <A class=dwbctl

href="http://www-106.ibm.com/developerworks/linux/">Linux</A>

<A

                 href="http://www-106.ibm.com/developerworks/"><IMG height=24 
                 alt=developerWorks 
                 src="Sorting files with sort and tsort_files/dwlogo-small.gif"

width=136 border=0></A>

<IMG height=1 alt=""

                 src="Sorting files with sort and tsort_files/c.gif" width=5

border=0>

<IMG height=1 alt=""

     src="Sorting files with sort and tsort_files/c.gif" width=1

border=0>

<A class=mainlink

     href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=http://www.ibm.com/ibm/?origin=dwheader">About 
     IBM</A>  |  <A 
     class=mainlink 
     href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=http://www.ibm.com/privacy/?origin=dwheader">Privacy</A>  |  <A class=mainlink 
     href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=http://www.ibm.com/legal/?origin=dwheader">Terms 
     of use</A>  |  <A 
     class=mainlink

href="http://www-106.ibm.com/developerworks/cgi-bin/click.cgi?url=http://www.ibm.com/contact/?origin=dwheader">Contact</A>

Sort and tsort file sorting in bash

Navigation menu

Search