SED用户手册

sed, a stream editor

Next: Introduction,Up: (dir)

sed, a stream editor

This file documents version 4.2.1 ofGNU sed, a stream editor.

Copyright © 1998, 1999, 2001, 2002, 2003, 2004 FreeSoftware Foundation, Inc.

This document is released under the terms of the GNU FreeDocumentation License as published by the Free Software Foundation;either version 1.1, or (at your option) any later version.

You should have received a copy of the GNU Free DocumentationLicense along withGNU sed; see the fileCOPYING.DOC. If not, write to the Free Software Foundation, 59 Temple Place - Suite330, Boston, MA 02110-1301, USA.

There are no Cover Texts and no Invariant Sections; this text, alongwith its equivalent in the printed manual, constitutes the Title Page.

• Introduction: Introduction
• Invoking sed: Invocation
• sed Programs: sed programs
• Examples: Some sample scripts
• Limitations: Limitations and (non-)limitations ofGNU sed
• Other Resources: Other resources for learning aboutsed
• Reporting Bugs: Reporting bugs
• Extended regexps: egrep-style regular expressions
• Concept Index: A menu with all the topics in this manual.
• Command and Option Index: A menu with allsed commands and command-line options.

--- The detailed node listing ---

sed Programs:

• Execution Cycle: How sed works
• Addresses: Selecting lines with sed
• Regular Expressions: Overview of regular expression syntax
• Common Commands: Often used commands
• The "s" Command: sed's Swiss Army Knife
• Other Commands: Less frequently used commands
• Programming Commands: Commands for sed gurus
• Extended Commands: Commands specific of GNU sed
• Escapes: Specifying special characters

Examples:

• Centering lines
• Increment a number
• Rename files to lower case
• Print bash environment
• Reverse chars of lines
• tac: Reverse lines of files
• cat -n: Numbering lines
• cat -b: Numbering non-blank lines
• wc -c: Counting chars
• wc -w: Counting words
• wc -l: Counting lines
• head: Printing the first lines
• tail: Printing the last lines
• uniq: Make duplicate lines unique
• uniq -d: Print duplicated lines of input
• uniq -u: Remove all duplicated lines
• cat -s: Squeezing blank lines

Next: Invoking sed,Previous: Top,Up: Top

1 Introduction

sed is a stream editor. A stream editor is used to perform basic texttransformations on an input stream(a file or input from a pipeline). While in some ways similar to an editor whichpermits scripted edits (such as ed),sedworks by making only one pass over theinput(s), and is consequently more efficient. But it issed's ability to filter text in a pipelinewhich particularly distinguishes it from other types ofeditors.

Next: sed Programs,Previous: Introduction,Up: Top

2 Invocation

Normally sed is invoked like this:

     sed SCRIPT INPUTFILE...

The full format for invoking sed is:

     sed OPTIONS... [SCRIPT] [INPUTFILE...]

If you do not specify INPUTFILE, or if INPUTFILE is -,sed filters the contents of the standard input. Thescriptis actually the first non-option parameter, which sed speciallyconsiders a script and not an input file if (and only if) none of theotheroptions specifies a script to be executed, that is if neitherof the -e and -f options is specified.

sed may be invoked with the following command-line options:

--version

Print out the version ofsed that is being run and a copyright notice,then exit.

--help

Print a usage message briefly summarizing these command-line optionsand the bug-reporting address,then exit.

-n

--quiet

--silent

By default,sed prints out the pattern spaceat the end of each cycle through the script (seeHow sedworks). These options disable this automatic printing,andsed only produces output when explicitly told tovia thep command. 

-e script

--expression=script

Add the commands inscript to the set of commands to berun while processing the input. 

-f script-file

--file=script-file

Add the commands contained in the filescript-fileto the set of commands to be run while processing the input.

-i[SUFFIX]

--in-place[=SUFFIX]

This option specifies that files are to be edited in-place. GNU sed does this by creating a temporary file andsending output to this file rather than to the standardoutput.¹.

This option implies -s.

When the end of the file is reached, the temporary file isrenamed to the output file's original name. The extension,if supplied, is used to modify the name of the old filebefore renaming the temporary file, thereby making a backupcopy²).

This rule is followed: if the extension doesn't contain a*,then it is appended to the end of the current filename as asuffix; if the extension does contain one or more*characters, then each asterisk is replaced with thecurrent filename. This allows you to add a prefix to thebackup file, instead of (or in addition to) a suffix, oreven to place backup copies of the original files into anotherdirectory (provided the directory already exists).

If no extension is supplied, the original file isoverwritten without making a backup.

-l N

--line-length=N

Specify the default line-wrap length for thel command. A length of 0 (zero) means to never wrap long lines. Ifnot specified, it is taken to be 70.

--posix

GNUsed includes several extensions to POSIXsed. In order to simplify writing portable scripts, thisoption disables all the extensions that this manual documents,including additional commands.Most of the extensions accept sed programs thatare outside the syntax mandated byPOSIX, but someof them (such as the behavior of the N commanddescribed in seeReporting Bugs) actually violate thestandard. If you want to disable only the latter kind ofextension, you can set thePOSIXLY_CORRECT variableto a non-empty value. 

-b

--binary

This option is available on every platform, but is only effective where theoperating system makes a distinction between text files and binary files. When such a distinction is made—as is the case for MS-DOS, Windows,Cygwin—text files are composed of lines separated by a carriage returnand a line feed character, andsed does not see theending CR. When this option is specified,sed will openinput files in binary mode, thus not requesting this special processingand considering lines to end at a line feed.

--follow-symlinks

This option is available only on platforms that supportsymbolic links and has an effect only if option-iis specified. In this case, if the file that is specifiedon the command line is a symbolic link,sed willfollow the link and edit the ultimate destination of thelink. The default behavior is to break the symbolic link,so that the link destination will not be modified.

-r

--regexp-extended

Use extended regular expressions rather than basicregular expressions. Extended regexps are those thategrep accepts; they can be clearer because theyusually have less backslashes, but are aGNUextensionand hence scripts that use them are not portable. SeeExtended regular expressions. 

-s

--separate

By default,sed will consider the files specified on thecommand line as a single continuous long stream. ThisGNU sedextension allows the user to consider them as separate files:range addresses (such as ‘/abc/,/def/’) are not allowedto span several files, line numbers are relative to the startof each file, $refers to the last line of each file,and files invoked from theR commands are rewound at thestart of each file. 

-u

--unbuffered

Buffer both input and output as minimally as practical. (This is particularly useful if the input is coming fromthe likes of ‘tail -f’, and you wish to see the transformedoutput as soon as possible.)

If no -e, -f,--expression, or --fileoptions are given on the command-line,then the first non-option argument on the command line istaken to be thescript to be executed.

If any command-line parameters remain after processing the above,these parameters are interpreted as the names of input files tobe processed.A file name of ‘-’ refers to the standard input stream. The standard input will be processed if no file names are specified.

Next: Examples,Previous: Invoking sed,Up: Top

3 sed Programs

Ased program consists of one or more sed commands,passed in by one or more of the-e,-f, --expression, and--fileoptions, or the first non-option argument if zero of theseoptions are used. This document will refer to “the”sed script;this is understood to mean the in-order catenationof all of thescripts and script-files passed in.

Commands within a script or script-file can beseparated by semicolons (;) or newlines (ASCII 10). Some commands, due to their syntax, cannot be followed by semicolonsworking as command separators and thus should be terminatedwith newlines or be placed at the end of a script or script-file. Commands can also be preceded with optional non-significantwhitespace characters.

Each sed command consists of an optional address oraddress range, followed by a one-character command nameand any additional command-specific code.

• Execution Cycle: How sed works
• Addresses: Selecting lines with sed
• Regular Expressions: Overview of regular expression syntax
• Common Commands: Often used commands
• The "s" Command: sed's Swiss Army Knife
• Other Commands: Less frequently used commands
• Programming Commands: Commands forsed gurus
• Extended Commands: Commands specific ofGNU sed
• Escapes: Specifying special characters

Next: Addresses,Up: sed Programs

3.1 How sed Works

sed maintains two data buffers: the active pattern space,and the auxiliary hold space. Both are initially empty.

sed operates by performing the following cycle on eachline of input: first,sed reads one line from the inputstream, removes any trailing newline, and places it in the pattern space. Then commands are executed; each command can have an address associatedto it: addresses are a kind of condition code, and a command is onlyexecuted if the condition is verified before the command is to beexecuted.

When the end of the script is reached, unless the -n optionis in use, the contents of pattern space are printed out to the outputstream, adding back the trailing newline if it was removed.³ Then the next cycle starts for the nextinput line.

Unless special commands (like ‘D’) are used, the pattern space isdeleted between two cycles. The hold space, on the other hand, keepsits data between cycles (see commands ‘h’, ‘H’, ‘x’,‘g’, ‘G’ to move data between both buffers).

Next: Regular Expressions,Previous: Execution Cycle,Up: sed Programs

3.2 Selecting lines with sed

Addresses in ased script can be in any of the following forms:

number

Specifying a line number will match only that line in the input. (Note thatsed counts lines continuously across all input filesunless-i or -s options are specified.)

first~step

ThisGNU extension matches every stepth linestarting with linefirst. In particular, lines will be selected when there existsa non-negativen such that the current line-number equalsfirst + (n *step). Thus, to select the odd-numbered lines,one would use 1~2;to pick every third line starting with the second, ‘2~3’ would be used;to pick every fifth line starting with the tenth, use ‘10~5’;and ‘50~0’ is just an obscure way of saying 50. 

$

This address matches the last line of the last file of input, orthe last line of each file when the-i or -soptionsare specified.

/regexp/

This will select any line which matches the regular expressionregexp. If regexp itself includes any /characters,each must be escaped by a backslash (\).

The empty regular expression ‘//’ repeats the last regularexpression match (the same holds if the empty regular expression ispassed to the s command). Note that modifiers to regular expressionsare evaluated when the regular expression is compiled, thus it is invalid tospecify them together with the empty regular expression. 

\%regexp%

(The % may be replaced by any other single character.)

This also matches the regular expressionregexp,but allows one to use a different delimiter than /. This is particularly useful if theregexp itself containsa lot of slashes, since it avoids the tedious escaping of every/. If regexp itself includes any delimiter characters,each must be escaped by a backslash (\).

/regexp/I

\%regexp%I

TheI modifier to regular-expression matching is a GNUextension which causes theregexp to be matched ina case-insensitive manner. 

/regexp/M

\%regexp%M

The M modifier to regular-expression matching is a GNU sedextension which causes ^ and $ to match respectively(in addition to the normal behavior) the empty string after a newline,and the empty string before a newline. There are special charactersequences(\` and\')which always match the beginning or the end of the buffer. M stands for multi-line.

If no addresses are given, then all lines are matched;if one address is given, then only lines matching thataddress are matched.

An address range can be specified by specifying two addressesseparated by a comma (,). An address range matches linesstarting from where the first address matches, and continuesuntil the second address matches (inclusively).

If the second address is a regexp, then checking for theending match will start with the linefollowing theline which matched the first address: a range will alwaysspan at least two lines (except of course if the input streamends).

If the second address is a number less than (or equal to)the line matching the first address, then only the one line ismatched.

GNUsed also supports some special two-address forms; all theseareGNU extensions:

0,/regexp/

A line number of 0 can be used in an address specification like0,/regexp/ so thatsed will try to matchregexp in the first input line too. In other words,0,/regexp/ is similar to1,/regexp/,except that ifaddr2 matches the very first line of input the0,/regexp/ form will consider it to end the range, whereasthe1,/regexp/ form will match the beginning of its range andhence make the range span up to thesecond occurrence of theregular expression.

Note that this is the only place where the 0 address makessense; there is no 0-th line and commands which are given the0address in any other way will give an error. 

addr1,+N

Matches addr1 and the N lines following addr1.

addr1,~N

Matches addr1 and the lines following addr1until the next line whose input line number is a multiple ofN.

Appending the! character to the end of an addressspecification negates the sense of the match. That is, if the! character follows an address range,then only lines which do not match the address rangewill be selected. This also works for singleton addresses,and, perhaps perversely, for the null address.

Next: Common Commands,Previous: Addresses,Up: sed Programs

3.3 Overview of Regular Expression Syntax

To know how to use sed, people should understand regularexpressions (regexp for short). A regular expressionis a pattern that is matched against asubject string from left to right. Most characters areordinary: they stand forthemselves in a pattern, and match the corresponding charactersin the subject. As a trivial example, the pattern

     The quick brown fox

matches a portion of a subject string that is identical toitself. The power of regular expressions comes from theability to include alternatives and repetitions in the pattern. These are encoded in the pattern by the use ofspecial characters,which do not stand for themselves but insteadare interpreted in some special way. Here is a brief descriptionof regular expression syntax as used insed.

char

A single ordinary character matches itself. 

*

Matches a sequence of zero or more instances of matches for thepreceding regular expression, which must be an ordinary character, aspecial character preceded by \, a ., a grouped regexp(see below), or a bracket expression. As aGNU extension, apostfixed regular expression can also be followed by*; forexample, a** is equivalent to a*. POSIX1003.1-2001 says that * stands for itself when it appears atthe start of a regular expression or subexpression, but manynonGNU implementations do not support this and portablescripts should instead use\* in these contexts. 

\+

As*, but matches one or more. It is a GNU extension.

\?

As*, but only matches zero or one. It is a GNU extension.

\{i\}

As *, but matches exactly i sequences (i is adecimal integer; for portability, keep it between 0 and 255inclusive).

\{i,j\}

Matches between i and j, inclusive, sequences. 

\{i,\}

Matches more than or equal to i sequences. 

\(regexp\)

Groups the inner regexp as a whole, this is used to:

·   Apply postfix operators, like\(abcd\)*:this will search for zero or more whole sequencesof ‘abcd’, whileabcd* would searchfor ‘abc’ followed by zero or more occurrencesof ‘d’. Note that support for\(abcd\)* isrequired by POSIX 1003.1-2001, but many non-GNUimplementations do not support it and hence it is not universallyportable.

·   Use back references (see below).

 

.

Matches any character, including newline. 

^

Matches the null string at beginning of the pattern space, i.e. whatappears after the circumflex must appear at the beginning of thepattern space.

In most scripts, pattern space is initialized to the content of eachline (see How sed works). So, it is auseful simplification to think of ^#include as matching onlylines where ‘#include’ is the first thing on line—if there arespaces before, for example, the match fails. This simplification isvalid as long as the original content of pattern space is not modified,for example with an s command.

^ acts as a special character only at the beginning of theregular expression or subexpression (that is, after\( or\|). Portable scripts should avoid ^ at the beginning ofa subexpression, though, asPOSIXallows implementations thattreat ^ as an ordinary character in that context.

$

It is the same as ^, but refers to end of pattern space. $ also acts as a special character only at the endof the regular expression or subexpression (that is, before\)or \|), and its use at the end of a subexpression is notportable.

[list]

[^list]

Matches any single character in list: for example,[aeiou] matches all vowels. A list may includesequences likechar1-char2, whichmatches any character between (inclusive)char1and char2.

A leading ^ reverses the meaning of list, so thatit matches any single characternot in list. To include] in the list, make it the first character (afterthe^ if needed), to include - in the list,make it the first or last; to include^ putit after the first character.

The characters$, *, ., [, and \are normally not special withinlist. For example, [\*]matches either ‘\’ or ‘*’, because the\ is notspecial here. However, strings like [.ch.], [=a=], and[:space:] are special withinlist and represent collatingsymbols, equivalence classes, and character classes, respectively, and[ is therefore special withinlist when it is followed by., =, or :. Also, when not inPOSIXLY_CORRECT mode, special escapes like\n and\t are recognized within list. See Escapes. 

regexp1\|regexp2

Matches eitherregexp1 or regexp2. Useparentheses to use complex alternative regular expressions. The matching process tries each alternative in turn, fromleft to right, and the first one that succeeds is used. It is aGNU extension. 

regexp1regexp2

Matches the concatenation of regexp1 and regexp2. Concatenation binds more tightly than\|, ^, and$, but less tightly than the other regular expressionoperators.

\digit

Matches the digit-th \(...\) parenthesizedsubexpression in the regular expression. This is called abackreference. Subexpressions are implicity numbered by countingoccurrences of\( left-to-right. 

\n

Matches the newline character. 

\char

Matches char, where char is one of $,*,., [, \, or ^. Note that the only C-likebackslash sequences that you can portably assume to beinterpreted are\n and \\; in particular\t is not portable, and matches a ‘t’ under mostimplementations ofsed, rather than a tab character.

Note that the regular expression matcher is greedy, i.e., matchesare attempted from left to right and, if two or more matches arepossible starting at the same character, it selects the longest.

Examples:

‘abcdef’

Matches ‘abcdef’. 

‘a*b’

Matches zero or more ‘a’s followed by a single‘b’. For example, ‘b’ or ‘aaaaab’.

‘a\?b’

Matches ‘b’ or ‘ab’.

‘a\+b\+’

Matches one or more ‘a’s followed by one or more‘b’s: ‘ab’ is the shortest possible match, butother examples are ‘aaaab’ or ‘abbbbb’ or‘aaaaaabbbbbbb’.

‘.*’

‘.\+’

These two both match all the characters in a string;however, the first matches every string (including the emptystring), while the second matches only strings containingat least one character.

‘^main.*(.*)’

This matches a string starting with ‘main’,followed by an opening and closingparenthesis. The ‘n’, ‘(’ and ‘)’ need notbe adjacent. 

‘^#’

This matches a string beginning with ‘#’.

‘\\$’

This matches a string ending with a single backslash. Theregexp contains two backslashes for escaping.

‘\$’

Instead, this matches a string consisting of a single dollar sign,because it is escaped.

‘[a-zA-Z0-9]’

In the C locale, this matches any ASCII letters or digits.

‘[^ tab]\+’

(Here tab stands for a single tab character.) This matches a string of one or morecharacters, none of which is a space or a tab. Usually this means a word.

‘^\(.*\)\n\1$’

This matches a string consisting of two equal substrings separated bya newline.

‘.\{9\}A$’

This matches nine characters followed by an ‘A’.

‘^.\{15\}A’

This matches the start of a string that contains 16 characters,the last of which is an ‘A’.

Next: The "s" Command,Previous: Regular Expressions,Up: sed Programs

3.4 Often-Used Commands

If you use sed at all, you will quite likely want to knowthese commands.

#

[No addresses allowed.]

The# character begins a comment;the comment continues until the next newline.

If you are concerned about portability, be aware thatsome implementations ofsed (which are notposixconformant) may only support a single one-line comment,and then only when the very first character of the script is a#.

Warning: if the first two characters of thesed scriptare #n, then the-n (no-autoprint) option is forced. If you want to put a comment in the first line of your scriptand that comment begins with the letter ‘n’and you do not want this behavior,then be sure to either use a capital ‘N’,or place at least one space before the ‘n’.

q [exit-code]

This command only accepts a single address.

Exitsed without processing any more commands or input. Note that the current pattern space is printed if auto-print isnot disabled with the-n options. The ability to returnan exit code from thesed script is a GNUsed extension. 

d

Delete the pattern space;immediately start next cycle.

p

Print out the pattern space (to the standard output). This command is usually only used in conjunction with the-ncommand-line option. 

n

If auto-print is not disabled, print the pattern space,then, regardless, replace the pattern space with the next line of input. If there is no more input then sed exits without processingany more commands.

{ commands }

A group of commands may be enclosed between{ and} characters. This is particularly useful when you want a group of commandsto be triggered by a single address (or address-range) match.

Next: Other Commands,Previous: Common Commands,Up: sed Programs

3.5 The s Command

The syntax of the s (as in substitute) command is‘s/regexp/replacement/flags’. The/characters may be uniformly replaced by any other singlecharacter within any givens command. The /character (or whatever other character is used in its stead)can appear in theregexp or replacementonly if it is preceded by a \character.

The s command is probably the most important in sedand has a lot of different options. Its basic concept is simple:thes command attempts to match the patternspace against the supplied regexp; if the match issuccessful, then that portion of the patternspace which was matched is replaced withreplacement.

Thereplacement can contain \n (n beinga number from 1 to 9, inclusive) references, which refer tothe portion of the match which is contained between thenth\( and its matching \). Also, the replacement can contain unescaped&characters which reference the whole matched portionof the pattern space.Finally, as a GNU sedextension, you can include aspecial sequence made of a backslash and one of the lettersL,l, U, u, or E. The meaning is as follows:

\L

Turn the replacementto lowercase until a \U or \E is found,

\l

Turn thenext character to lowercase, 

\U

Turn the replacement to uppercaseuntil a \L or \E is found,

\u

Turn the next characterto uppercase, 

\E

Stop case conversion started by \L or \U.

To include a literal \, &, or newline in the finalreplacement, be sure to precede the desired\, &,or newline in thereplacement with a \.

Thes command can be followed by zero or more of thefollowing flags:

g

Apply the replacement toall matches to the regexp,not just the first. 

number

Only replace thenumberth match of the regexp.

Note: the posix standard does not specify what should happenwhen you mix theg and numbermodifiers,and currently there is no widely agreed upon meaningacrosssed implementations. ForGNUsed, the interaction is defined to be:ignore matches before thenumberth,and then match and replace all matches fromthe numberth on.

p

If the substitution was made, then print the new pattern space.

Note: when both the p and e options are specified,the relative ordering of the two produces very different results. In general,ep (evaluate then print) is what you want,but operating the other way round can be useful for debugging. For this reason, the current version ofGNU sed interpretsspecially the presence ofp options both before and aftere, printing the pattern space before and after evaluation,while in general flags for thes command show theireffect just once. This behavior, although documented, mightchange in future versions.

w file-name

If the substitution was made, then write out the result to the named file. As a GNU sed extension, two special values offile-name aresupported: /dev/stderr, which writes the result to the standarderror, and/dev/stdout, which writes to the standardoutput.⁴

e

This command allows one to pipe input from a shell commandinto pattern space. If a substitution was made, the commandthat is found in pattern space is executed and pattern spaceis replaced with its output. A trailing newline is suppressed;results are undefined if the command to be executed containsanul character. This is aGNU sed extension. 

I

i

TheI modifier to regular-expression matching is a GNUextension which makessed match regexp in acase-insensitive manner.

M

m

TheM modifier to regular-expression matching is a GNUsedextension which causes ^ and$ to match respectively(in addition to the normal behavior) the empty string after a newline,and the empty string before a newline. There are special charactersequences(\` and\')which always match the beginning or the end of the buffer. M stands for multi-line.

Next: Programming Commands,Previous: The "s" Command,Up: sed Programs

3.6 Less Frequently-Used Commands

Though perhaps less frequently used than those in the previoussection, some very small yet usefulsed scripts can be built withthese commands.

y/source-chars/dest-chars/

(The / characters may be uniformly replaced byany other single character within any giveny command.)

Transliterate any characters in the pattern space which matchany of thesource-chars with the corresponding characterin dest-chars.

Instances of the / (or whatever other character is used in its stead),\, or newlines can appear in thesource-chars or dest-charslists, provide that each instance is escaped by a\. The source-chars anddest-chars lists mustcontain the same number of characters (after de-escaping).

a\

text

As aGNU extension, this command accepts two addresses.

Queue the lines of text which follow this command(each but the last ending with a\,which are removed from the output)to be output at the end of the current cycle,or when the next input line is read.

Escape sequences in text are processed, so you shoulduse \\ intext to print a single backslash.

As a GNU extension, if between the a and the newline there isother than a whitespace-\ sequence, then the text of this line,starting at the first non-whitespace character after thea,is taken as the first line of thetext block. (This enables a simplification in scripting a one-line add.) This extension also works with theiand c commands. 

i\

text

As aGNU extension, this command accepts two addresses.

Immediately output the lines of text which follow this command(each but the last ending with a\,which are removed from the output). 

c\

text

Delete the lines matching the address or address-range,and output the lines of text which follow this command(each but the last ending with a \,which are removed from the output)in place of the last line(or in place of each line, if no addresses were specified). A new cycle is started after this command is done,since the pattern space will have been deleted.

=

As aGNU extension, this command accepts two addresses.

Print out the current input line number (with a trailing newline).

l n

Print the pattern space in an unambiguous form:non-printable characters (and the \ character)are printed in C-style escaped form; long lines are split,with a trailing\ character to indicate the split;the end of each line is marked with a$.

n specifies the desired line-wrap length;a length of 0 (zero) means to never wrap long lines. If omitted,the default as specified on the command line is used. Thenparameter is a GNU sed extension.

r filename

As aGNU extension, this command accepts two addresses.

Queue the contents offilename to be read andinserted into the output stream at the end of the current cycle,or when the next input line is read. Note that iffilename cannot be read, it is treated asif it were an empty file, without any error indication.

As a GNU sed extension, the special value/dev/stdinis supported for the file name, which reads the contents of thestandard input.

w filename

Write the pattern space to filename. As a GNU sed extension, two special values offile-namearesupported: /dev/stderr, which writes the result to the standarderror, and/dev/stdout, which writes to the standardoutput.⁵

The file will be created (or truncated) before the first input line isread; allw commands (including instances of the w flagon successfuls commands) which refer to the same filenameare output without closing and reopening the file.

D

If pattern space contains no newline, start a normal new cycle as ifthed command was issued. Otherwise, delete text in the patternspace up to the first newline, and restart cycle with the resultantpattern space, without reading a new line of input.

N

Add a newline to the pattern space,then append the next line of input to the pattern space. If there is no more input then sed exits without processingany more commands.

P

Print out the portion of the pattern space up to the first newline.

h

Replace the contents of the hold space with the contents of the pattern space. 

H

Append a newline to the contents of the hold space,and then append the contents of the pattern space to that of the hold space. 

g

Replace the contents of the pattern space with the contents of the hold space. 

G

Append a newline to the contents of the pattern space,and then append the contents of the hold space to that of the pattern space. 

x

Exchange the contents of the hold and pattern spaces.

Next: Extended Commands,Previous: Other Commands,Up: sed Programs

3.7 Commands for sed gurus

In most cases, use of these commands indicates that you areprobably better off programming in something likeawkor Perl. But occasionally one is committed to stickingwithsed, and these commands can enable one to writequite convoluted scripts.

: label

[No addresses allowed.]

Specify the location oflabel for branch commands. In all other respects, a no-op. 

b label

Unconditionally branch tolabel. The label may be omitted, in which case the next cycle is started.

t label

Branch tolabel only if there has been a successful substitutionsince the last input line was read or conditional branch was taken. Thelabel may be omitted, in which case the next cycle is started.

Next: Escapes,Previous: Programming Commands,Up: sed Programs

3.8 Commands Specific to GNU sed

These commands are specific to GNU sed, so youmust use them with care and only when you are sure thathindering portability is not evil. They allow you to checkforGNU sed extensions or to do tasks that are requiredquite often, yet are unsupported by standardseds.

e [command]

This command allows one to pipe input from a shell commandinto pattern space. Without parameters, thee commandexecutes the command that is found in pattern space andreplaces the pattern space with the output; a trailing newlineis suppressed.

If a parameter is specified, instead, the e commandinterprets it as a command and sends its output to the output stream(liker does). The command can run across multiplelines, all but the last ending with a back-slash.

In both cases, the results are undefined if the command to beexecuted contains anul character. 

F

Print out the file name of the current input file (with a trailingnewline).

L n

ThisGNU sed extension fills and joins lines in pattern spaceto produce output lines of (at most)ncharacters, likefmt does; if n is omitted, the default as specifiedon the command line is used. This command is considered a failedexperiment and unless there is enough request (which seems unlikely)will be removed in future versions. 

Q [exit-code]

This command only accepts a single address.

This command is the same as q, but will not print thecontents of pattern space. Likeq, it provides theability to return an exit code to the caller.

This command can be useful because the only alternative waysto accomplish this apparently trivial function are to usethe-n option (which can unnecessarily complicateyour script) or resorting to the following snippet, whichwastes time by reading the whole file without any visible effect:

          :eat

          $d       Quit silently on the last line

          N        Read another line, silently

          g        Overwrite pattern space each time to save memory

          b eat

 

R filename

Queue a line of filename to be read andinserted into the output stream at the end of the current cycle,or when the next input line is read. Note that iffilename cannot be read, or if its end isreached, no line is appended, without any error indication.

As with the r command, the special value /dev/stdinis supported for the file name, which reads a line from thestandard input.

T label

Branch to label only if there have been no successfulsubstitutions since the last input line was read orconditional branch was taken. Thelabel may be omitted,in which case the next cycle is started. 

v version

This command does nothing, but makessed fail ifGNU sed extensions are not supported, simply because otherversions ofsed do not implement it. In addition, youcan specify the version ofsed that your scriptrequires, such as 4.0.5. The default is 4.0because that is the first version that implemented this command.

This command enables all GNU extensions even ifPOSIXLY_CORRECT is set in the environment.

W filename

Write to the given filename the portion of the pattern space up tothe first newline. Everything said under the w command aboutfile handling holds here too.

z

This command empties the content of pattern space. It isusually the same as ‘s/.*//’, but is more efficientand works in the presence of invalid multibyte sequencesin the input stream.posix mandates that such sequencesare not matched by ‘.’, so that there is no portableway to clearsed's buffers in the middle of thescript in most multibyte locales (including UTF-8 locales).

Previous: Extended Commands,Up: sed Programs

3.9 GNU Extensions for Escapes in Regular Expressions

Until this chapter, we have only encountered escapes of the form‘\^’, which tellsed not to interpret the circumflexas a special character, but rather to take it literally. Forexample, ‘\*’ matches a single asterisk rather than zeroor more backslashes.

This chapter introduces another kind of escape⁶—thatis, escapes that are applied to a character or sequence of charactersthat ordinarily are taken literally, and that sed replaceswith a special character. This provides a wayof encoding non-printable characters in patterns in a visible manner. There is no restriction on the appearance of non-printing charactersin a sed script but when a script is being prepared in theshell or by text editing, it is usually easier to use one ofthe following escape sequences than the binary character itrepresents:

The list of these escapes is:

\a

Produces or matches a bel character, that is an “alert” (ascii 7).

\f

Produces or matches a form feed (ascii 12). 

\n

Produces or matches a newline (ascii 10). 

\r

Produces or matches a carriage return (ascii 13). 

\t

Produces or matches a horizontal tab (ascii 9). 

\v

Produces or matches a so called “vertical tab” (ascii 11).

\cx

Produces or matches Control-x, wherex isany character. The precise effect of ‘\cx’ is as follows:ifx is a lower case letter, it is converted to upper case. Then bit 6 of the character (hex 40) is inverted. Thus ‘\cz’ becomeshex 1A, but ‘\c{’ becomes hex 3B, while ‘\c;’ becomes hex 7B. 

\dxxx

Produces or matches a character whose decimal ascii value isxxx. 

\oxxx

Produces or matches a character whose octal ascii value isxxx. 

\xxx

Produces or matches a character whose hexadecimal ascii value isxx.

‘\b’ (backspace) was omitted because of the conflict withthe existing “word boundary” meaning.

Other escapes match a particular character class and are valid only inregular expressions:

\w

Matches any “word” character. A “word” character is anyletter or digit or the underscore character.

\W

Matches any “non-word” character. 

\b

Matches a word boundary; that is it matches if the characterto the left is a “word” character and the character to theright is a “non-word” character, or vice-versa.

\B

Matches everywhere but on a word boundary; that is it matchesif the character to the left and the character to the rightare either both “word” characters or both “non-word”characters.

\`

Matches only at the start of pattern space. This is differentfrom ^ in multi-line mode.

\'

Matches only at the end of pattern space. This is differentfrom $ in multi-line mode.

Next: Limitations,Previous: sed Programs,Up: Top

4 Some Sample Scripts

Here are some sed scripts to guide you in the art of masteringsed.

Some exotic examples:

• Centering lines
• Increment a number
• Rename files to lower case
• Print bash environment
• Reverse chars of lines

Emulating standard utilities:

• tac: Reverse lines of files
• cat -n: Numbering lines
• cat -b: Numbering non-blank lines
• wc -c: Counting chars
• wc -w: Counting words
• wc -l: Counting lines
• head: Printing the first lines
• tail: Printing the last lines
• uniq: Make duplicate lines unique
• uniq -d: Print duplicated lines of input
• uniq -u: Remove all duplicated lines
• cat -s: Squeezing blank lines

Next: Increment a number,Up: Examples

4.1 Centering Lines

This script centers all lines of a file on a 80 columns width. To change that width, the number in\{...\} must bereplaced, and the number of added spaces also must be changed.

Note how the buffer commands are used to separate parts inthe regular expressions to be matched—this is a commontechnique.

     #!/usr/bin/sed -f

    

     # Put 80 spaces in the buffer

     1 {

       x

       s/^$/          /

       s/^.*$/&&&&&&&&/

       x

     }

    

     # del leading and trailing spaces

     y/tab/ /

     s/^ *//

     s/ *$//

    

     # add a newline and 80 spaces to end of line

     G

    

     # keep first 81 chars (80 + a newline)

     s/^\(.\{81\}\).*$/\1/

    

     # \2 matches half of the spaces, which are moved to the beginning

     s/^\(.*\)\n\(.*\)\2/\2\1/

Next: Rename files to lower case,Previous: Centering lines,Up: Examples

4.2 Increment a Number

This script is one of a few that demonstrate how to do arithmeticin sed. This is indeed possible,⁷ but must be done manually.

To increment one number you just add 1 to last digit, replacingit by the following digit. There is one exception: when the digitis a nine the previous digits must be also incremented until youdon't have a nine.

This solution by Bruno Haible is very clever and smart becauseit uses a single buffer; if you don't have this limitation, thealgorithm used inNumbering lines, is faster. It works by replacing trailing nines with an underscore, thenusing multiples commands to increment the last digit,and then again substituting underscores with zeros.

     #!/usr/bin/sed -f

    

     /[^0-9]/ d

    

     # replace all leading 9s by _ (any other character except digits, could

     # be used)

     :d

     s/9\(_*\)$/_\1/

     td

    

     # incr last digit only.  The first line adds a most-significant

     # digit of 1 if we have to add a digit.

     #

     # The tn commands are not necessary, but make the thing

     # faster

    

     s/^\(_*\)$/1\1/; tn

     s/8\(_*\)$/9\1/; tn

     s/7\(_*\)$/8\1/; tn

     s/6\(_*\)$/7\1/; tn

     s/5\(_*\)$/6\1/; tn

     s/4\(_*\)$/5\1/; tn

     s/3\(_*\)$/4\1/; tn

     s/2\(_*\)$/3\1/; tn

     s/1\(_*\)$/2\1/; tn

     s/0\(_*\)$/1\1/; tn

    

     :n

     y/_/0/

Next: Print bash environment,Previous: Increment a number,Up: Examples

4.3 Rename Files to Lower Case

This is a pretty strange use of sed. We transform text, andtransform it to be shell commands, then just feed them to shell. Don't worry, even worse hacks are done when usingsed; I haveseen a script converting the output ofdate into a bcprogram!

The main body of this is the sed script, which remaps the namefrom lower to upper (or vice-versa) and even checks outif the remapped name is the same as the original name. Note how the script is parameterized using shellvariables and proper quoting.

     #! /bin/sh

     # rename files to lower/upper case...

     #

     # usage:

     #    move-to-lower *

     #    move-to-upper *

     # or

     #    move-to-lower -R .

     #    move-to-upper -R .

     #

    

     help()

     {

             cat << eof

     Usage: $0 [-n] [-r] [-h] files...

    

     -n      do nothing, only see what would be done

     -R      recursive (use find)

     -h      this message

     files   files to remap to lower case

    

     Examples:

            $0 -n *        (see if everything is ok, then...)

            $0 *

    

            $0 -R .

    

     eof

     }

    

     apply_cmd='sh'

     finder='echo "$@" | tr " " "\n"'

     files_only=

    

     while :

     do

         case "$1" in

             -n) apply_cmd='cat' ;;

             -R) finder='find "$@" -type f';;

             -h) help ; exit 1 ;;

             *) break ;;

         esac

         shift

     done

    

     if [ -z "$1" ]; then

             echo Usage: $0 [-h] [-n] [-r] files...

             exit 1

     fi

    

     LOWER='abcdefghijklmnopqrstuvwxyz'

     UPPER='ABCDEFGHIJKLMNOPQRSTUVWXYZ'

    

     case `basename $0` in

             *upper*) TO=$UPPER; FROM=$LOWER ;;

             *)       FROM=$UPPER; TO=$LOWER ;;

     esac

    

     eval $finder | sed -n '

    

     # remove all trailing slashes

     s/\/*$//

    

     # add ./ if there is no path, only a filename

     /\//! s/^/.\//

    

     # save path+filename

     h

    

     # remove path

     s/.*\///

    

     # do conversion only on filename

     y/'$FROM'/'$TO'/

    

     # now line contains original path+file, while

     # hold space contains the new filename

     x

    

     # add converted file name to line, which now contains

     # path/file-name\nconverted-file-name

     G

    

     # check if converted file name is equal to original file name,

     # if it is, do not print nothing

     /^.*\/\(.*\)\n\1/b

    

     # now, transform path/fromfile\n, into

     # mv path/fromfile path/tofile and print it

     s/^\(.*\/\)\(.*\)\n\(.*\)$/mv "\1\2" "\1\3"/p

    

     ' | $apply_cmd

Next: Reverse chars of lines,Previous: Rename files to lower case,Up: Examples

4.4 Print bash Environment

This script strips the definition of the shell functionsfrom the output of theset Bourne-shell command.

     #!/bin/sh

    

     set | sed -n '

     :x

    

     # if no occurrence of ‘=()’ print and load next line

     /=()/! { p; b; }

     / () $/! { p; b; }

    

     # possible start of functions section

     # save the line in case this is a var like FOO="() "

     h

    

     # if the next line has a brace, we quit because

     # nothing comes after functions

     n

     /^{/ q

    

     # print the old line

     x; p

    

     # work on the new line now

     x; bx

     '

Next: tac,Previous: Print bash environment,Up: Examples

4.5 Reverse Characters of Lines

This script can be used to reverse the position of charactersin lines. The technique moves two characters at a time, henceit is faster than more intuitive implementations.

Note the tx command before the definition of the label. This is often needed to reset the flag that is tested bythet command.

Imaginative readers will find uses for this script. An exampleis reversing the output ofbanner.⁸

     #!/usr/bin/sed -f

    

     /../! b

    

     # Reverse a line.  Begin embedding the line between two newlines

     s/^.*$/\

     &\

     /

    

     # Move first character at the end.  The regexp matches until

     # there are zero or one characters between the markers

     tx

     :x

     s/\(\n.\)\(.*\)\(.\n\)/\3\2\1/

     tx

    

     # Remove the newline markers

     s/\n//g

Next: cat -n,Previous: Reverse chars of lines,Up: Examples

4.6 Reverse Lines of Files

This one begins a series of totally useless (yet interesting)scripts emulating various Unix commands. This, in particular,is atac workalike.

Note that on implementations other than GNU sedthis script might easily overflow internal buffers.

     #!/usr/bin/sed -nf

    

     # reverse all lines of input, i.e. first line became last, ...

    

     # from the second line, the buffer (which contains all previous lines)

     # is *appended* to current line, so, the order will be reversed

     1! G

    

     # on the last line we're done -- print everything

     $ p

    

     # store everything on the buffer again

     h

Next: cat -b,Previous: tac,Up: Examples

4.7 Numbering Lines

This script replaces ‘cat -n’; in fact it formats its outputexactly likeGNU cat does.

Of course this is completely useless and for two reasons: first,because somebody else did it in C, second, because the followingBourne-shell script could be used for the same purpose and wouldbe much faster:

     #! /bin/sh

     sed -e "=" $@ | sed -e '

       s/^/      /

       N

       s/^ *\(......\)\n/\1  /

     '

It uses sed to print the line number, then groups lines twoby two usingN. Of course, this script does not teach as much asthe one presented below.

The algorithm used for incrementing uses both buffers, so the lineis printed as soon as possible and then discarded. The numberis split so that changing digits go in a buffer and unchanged ones goin the other; the changed digits are modified in a single step(using a y command). The line number for the next lineis then composed and stored in the hold space, to be used in thenext iteration.

     #!/usr/bin/sed -nf

    

     # Prime the pump on the first line

     x

     /^$/ s/^.*$/1/

    

     # Add the correct line number before the pattern

     G

     h

    

     # Format it and print it

     s/^/      /

     s/^ *\(......\)\n/\1  /p

    

     # Get the line number from hold space; add a zero

     # if we're going to add a digit on the next line

     g

     s/\n.*$//

     /^9*$/ s/^/0/

    

     # separate changing/unchanged digits with an x

     s/.9*$/x&/

    

     # keep changing digits in hold space

     h

     s/^.*x//

     y/0123456789/1234567890/

     x

    

     # keep unchanged digits in pattern space

     s/x.*$//

    

     # compose the new number, remove the newline implicitly added by G

     G

     s/\n//

     h

Next: wc -c,Previous: cat -n,Up: Examples

4.8 Numbering Non-blank Lines

Emulating ‘cat -b’ is almost the same as ‘cat -n’—we onlyhave to select which lines are to be numbered and which are not.

The part that is common to this script and the previous one isnot commented to show how important it is to commentsedscripts properly...

     #!/usr/bin/sed -nf

    

     /^$/ {

       p

       b

     }

    

     # Same as cat -n from now

     x

     /^$/ s/^.*$/1/

     G

     h

     s/^/      /

     s/^ *\(......\)\n/\1  /p

     x

     s/\n.*$//

     /^9*$/ s/^/0/

     s/.9*$/x&/

     h

     s/^.*x//

     y/0123456789/1234567890/

     x

     s/x.*$//

     G

     s/\n//

     h

Next: wc -w,Previous: cat -b,Up: Examples

4.9 Counting Characters

This script shows another way to do arithmetic with sed. In this case we have to add possibly large numbers, so implementingthis by successive increments would not be feasible (and possiblyeven more complicated to contrive than this script).

The approach is to map numbers to letters, kind of an abacusimplemented with sed. ‘a’s are units, ‘b’s aretens and so on: we simply add the number of characterson the current line as units, and then propagate the carryto tens, hundreds, and so on.

As usual, running totals are kept in hold space.

On the last line, we convert the abacus form back to decimal. For the sake of variety, this is done with a loop rather thanwith some 80s commands⁹: first weconvert units, removing ‘a’s from the number; then werotate letters so that tens become ‘a’s, and so onuntil no more letters remain.

     #!/usr/bin/sed -nf

    

     # Add n+1 a's to hold space (+1 is for the newline)

     s/./a/g

     H

     x

     s/\n/a/

    

     # Do the carry.  The t's and b's are not necessary,

     # but they do speed up the thing

     t a

     : a;  s/aaaaaaaaaa/b/g; t b; b done

     : b;  s/bbbbbbbbbb/c/g; t c; b done

     : c;  s/cccccccccc/d/g; t d; b done

     : d;  s/dddddddddd/e/g; t e; b done

     : e;  s/eeeeeeeeee/f/g; t f; b done

     : f;  s/ffffffffff/g/g; t g; b done

     : g;  s/gggggggggg/h/g; t h; b done

     : h;  s/hhhhhhhhhh//g

    

     : done

     $! {

       h

       b

     }

    

     # On the last line, convert back to decimal

    

     : loop

     /a/! s/[b-h]*/&0/

     s/aaaaaaaaa/9/

     s/aaaaaaaa/8/

     s/aaaaaaa/7/

     s/aaaaaa/6/

     s/aaaaa/5/

     s/aaaa/4/

     s/aaa/3/

     s/aa/2/

     s/a/1/

    

     : next

     y/bcdefgh/abcdefg/

     /[a-h]/ b loop

     p

Next: wc -l,Previous: wc -c,Up: Examples

4.10 Counting Words

This script is almost the same as the previous one, once eachof the words on the line is converted to a single ‘a’(in the previous script each letter was changed to an ‘a’).

It is interesting that real wc programs have optimizedloops for ‘wc -c’, so they are much slower at countingwords rather than characters. This script's bottleneck,instead, is arithmetic, and hence the word-counting oneis faster (it has to manage smaller numbers).

Again, the common parts are not commented to show the importanceof commenting sed scripts.

     #!/usr/bin/sed -nf

    

     # Convert words to a's

     s/[ tab][ tab]*/ /g

     s/^/ /

     s/ [^ ][^ ]*/a /g

     s/ //g

    

     # Append them to hold space

     H

     x

     s/\n//

    

     # From here on it is the same as in wc -c.

     /aaaaaaaaaa/! bx;   s/aaaaaaaaaa/b/g

     /bbbbbbbbbb/! bx;   s/bbbbbbbbbb/c/g

     /cccccccccc/! bx;   s/cccccccccc/d/g

     /dddddddddd/! bx;   s/dddddddddd/e/g

     /eeeeeeeeee/! bx;   s/eeeeeeeeee/f/g

     /ffffffffff/! bx;   s/ffffffffff/g/g

     /gggggggggg/! bx;   s/gggggggggg/h/g

     s/hhhhhhhhhh//g

     :x

     $! { h; b; }

     :y

     /a/! s/[b-h]*/&0/

     s/aaaaaaaaa/9/

     s/aaaaaaaa/8/

     s/aaaaaaa/7/

     s/aaaaaa/6/

     s/aaaaa/5/

     s/aaaa/4/

     s/aaa/3/

     s/aa/2/

     s/a/1/

     y/bcdefgh/abcdefg/

     /[a-h]/ by

     p

Next: head,Previous: wc -w,Up: Examples

4.11 Counting Lines

No strange things are done now, because sed gives us‘wc -l’ functionality for free!!! Look:

     #!/usr/bin/sed -nf

     $=

Next: tail,Previous: wc -l,Up: Examples

4.12 Printing the First Lines

This script is probably the simplest useful sed script. It displays the first 10 lines of input; the number of displayedlines is right before theq command.

     #!/usr/bin/sed -f

     10q

Next: uniq,Previous: head,Up: Examples

4.13 Printing the Last Lines

Printing the last n lines rather than the first is more complexbut indeed possible.n is encoded in the second line, beforethe bang character.

This script is similar to the tac script in that it keeps thefinal output in the hold space and prints it at the end:

     #!/usr/bin/sed -nf

    

     1! {; H; g; }

     1,10 !s/[^\n]*\n//

     $p

     h

Mainly, the scripts keeps a window of 10 lines and slides itby adding a line and deleting the oldest (the substitution commandon the second line works like aD command but does notrestart the loop).

The “sliding window” technique is a very powerful way to writeefficient and complexsed scripts, because commands likeP would require a lot of work if implemented manually.

To introduce the technique, which is fully demonstrated in therest of this chapter and is based on theN, Pand Dcommands, here is an implementation oftailusing a simple “sliding window.”

This looks complicated but in fact the working is the same asthe last script: after we have kicked in the appropriate numberof lines, however, we stop using the hold space to keep inter-linestate, and instead useNand D to slide patternspace by one line:

     #!/usr/bin/sed -f

    

     1h

     2,10 {; H; g; }

     $q

     1,9d

     N

     D

Note how the first, second and fourth line are inactive afterthe first ten lines of input. After that, all the script doesis: exiting on the last line of input, appending the next inputline to pattern space, and removing the first line.

Next: uniq -d,Previous: tail,Up: Examples

4.14 Make Duplicate Lines Unique

This is an example of the art of using the N, Pand D commands, probably the most difficult to master.

     #!/usr/bin/sed -f

     h

    

     :b

     # On the last line, print and exit

     $b

     N

     /^\(.*\)\n\1$/ {

         # The two lines are identical.  Undo the effect of

         # the n command.

         g

         bb

     }

    

     # If the N command had added the last line, print and exit

     $b

     

     # The lines are different; print the first and go

     # back working on the second.

     P

     D

As you can see, we mantain a 2-line window using P and D. This technique is often used in advancedsedscripts.

Next: uniq -u,Previous: uniq,Up: Examples

4.15 Print Duplicated Lines of Input

This script prints only duplicated lines, like ‘uniq -d’.

     #!/usr/bin/sed -nf

    

     $b

     N

     /^\(.*\)\n\1$/ {

         # Print the first of the duplicated lines

         s/.*\n//

         p

    

         # Loop until we get a different line

         :b

         $b

         N

         /^\(.*\)\n\1$/ {

             s/.*\n//

             bb

         }

     }

    

     # The last line cannot be followed by duplicates

     $b

    

     # Found a different one.  Leave it alone in the pattern space

     # and go back to the top, hunting its duplicates

     D

Next: cat -s,Previous: uniq -d,Up: Examples

4.16 Remove All Duplicated Lines

This script prints only unique lines, like ‘uniq -u’.

     #!/usr/bin/sed -f

    

     # Search for a duplicate line --- until that, print what you find.

     $b

     N

     /^\(.*\)\n\1$/ ! {

         P

         D

     }

    

     :c

     # Got two equal lines in pattern space.  At the

     # end of the file we simply exit

     $d

    

     # Else, we keep reading lines with N until we

     # find a different one

     s/.*\n//

     N

     /^\(.*\)\n\1$/ {

         bc

     }

    

     # Remove the last instance of the duplicate line

     # and go back to the top

     D

Previous: uniq -u,Up: Examples

4.17 Squeezing Blank Lines

As a final example, here are three scripts, of increasing complexityand speed, that implement the same function as ‘cat -s’, that issqueezing blank lines.

The first leaves a blank line at the beginning and end if there aresome already.

     #!/usr/bin/sed -f

    

     # on empty lines, join with next

     # Note there is a star in the regexp

     :x

     /^\n*$/ {

     N

     bx

     }

    

     # now, squeeze all '\n', this can be also done by:

     # s/^\(\n\)*/\1/

     s/\n*/\

     /

This one is a bit more complex and removes all empty linesat the beginning. It does leave a single blank line at endif one was there.

     #!/usr/bin/sed -f

    

     # delete all leading empty lines

     1,/^./{

     /./!d

     }

    

     # on an empty line we remove it and all the following

     # empty lines, but one

     :x

     /./!{

     N

     s/^\n$//

     tx

     }

This removes leading and trailing blank lines. It is also thefastest. Note that loops are completely done withnandb, without relying on sed to restart thethe script automatically at the end of a line.

     #!/usr/bin/sed -nf

    

     # delete all (leading) blanks

     /./!d

    

     # get here: so there is a non empty

     :x

     # print it

     p

     # get next

     n

     # got chars? print it again, etc...

     /./bx

    

     # no, don't have chars: got an empty line

     :z

     # get next, if last line we finish here so no trailing

     # empty lines are written

     n

     # also empty? then ignore it, and get next... this will

     # remove ALL empty lines

     /./!bz

    

     # all empty lines were deleted/ignored, but we have a non empty.  As

     # what we want to do is to squeeze, insert a blank line artificially

     i\

    

     bx

Next: Other Resources,Previous: Examples,Up: Top

5 GNU sed's Limitations and Non-limitations

For those who want to write portablesed scripts,be aware that some implementations have been known tolimit line lengths (for the pattern and hold spaces)to be no more than 4000 bytes. Theposix standard specifies that conforming sedimplementations shall support at least 8192 byte line lengths.GNU sed has no built-in limit on line length;as long as it canmalloc() more (virtual) memory,you can feed or construct lines as long as you like.

However, recursion is used to handle subpatterns and indefiniterepetition. This means that the available stack space may limitthe size of the buffer that can be processed by certain patterns.

Next: Reporting Bugs,Previous: Limitations,Up: Top

6 Other Resources for Learning About sed

In addition to several books that have been written aboutsed(either specifically or as chapters in books which discussshell programming), one can find out more aboutsed(including suggestions of a few books) from the FAQfor thesed-users mailing list, available from:

    

Also of interest areand,which include sed tutorials and other sed-related goodies.

The sed-users mailing list itself maintained by Sven Guckes. To subscribe, visit and searchfor the sed-users mailing list.

Next: Extended regexps,Previous: Other Resources,Up: Top

7 Reporting Bugs

Email bug reports to . Be sure to include the word “sed” somewhere in the Subject: field. Also, please include the output of ‘sed --version’ in the bodyof your report if at all possible.

Please do not send a bug report like this:

     while building frobme-1.3.4

     $ configure

     error--> sed: file sedscr line 1: Unknown option to 's'

If GNU sed doesn't configure your favorite package, take afew extra minutes to identify the specific problem and make a stand-alonetest case. Unlike other programs such as C compilers, making such testcases for sedis quite simple.

A stand-alone test case includes all the data necessary to perform thetest, and the specific invocation ofsedthat causes the problem. The smaller a stand-alone test case is, the better. A test case shouldnot involve something as far removed fromsed as “try to configurefrobme-1.3.4”. Yes, that is in principle enough information to lookfor the bug, but that is not a very practical prospect.

Here are a few commonly reported bugs that are not bugs.

N command on the last line

Most versions ofsed exit without printing anything whentheN command is issued on the last line of a file.GNU sed prints pattern space before exiting unless of coursethe-n command switch has been specified. This choice isby design.

For example, the behavior of

          sed N foo bar

would depend on whether foo has an even or an odd number oflines¹⁰. Or, when writing a script to read thenext few lines following a pattern match, traditionalimplementations ofsed would force you to writesomething like

          /foo/{ $!N; $!N; $!N; $!N; $!N; $!N; $!N; $!N; $!N }

instead of just

          /foo/{ N;N;N;N;N;N;N;N;N; }

In any case, the simplest workaround is to use$d;N inscripts that rely on the traditional behavior, or to setthe POSIXLY_CORRECT variable to a non-empty value. 

Regex syntax clashes (problems with backslashes)

sed uses theposix basic regular expression syntax. According tothe standard, the meaning of some escape sequences is undefined inthis syntax; notable in the case ofsed are \|,\+,\?, \`, \', \<,\>,\b, \B,\w, and \W.

As in all GNU programs that use posix basic regularexpressions,sed interprets these escape sequences as specialcharacters. So,x\+ matches one or more occurrences of ‘x’.abc\|def matches either ‘abc’ or ‘def’.

This syntax may cause problems when running scripts written for otherseds. Somesed programs have been written with theassumption that\| and \+ match the literal characters| and+. Such scripts must be modified by removing thespurious backslashes if they are to be used with modern implementationsofsed, likeGNU sed.

On the other hand, some scripts use s|abc\|def||g to remove occurrencesof either abc or def. While this worked untilsed 4.0.x, newer versions interpret this as removing thestringabc|def. This is again undefined behavior according toPOSIX, and this interpretation is arguably more robust: olderseds, for example, required that the regex matcher parsed\/ as / in the common case of escaping a slash, which isagain undefined behavior; the new behavior avoids this, and this is goodbecause the regex matcher is only partially under our control.

In addition, this version ofsed supports several escape characters(some of which are multi-character) to insert non-printable charactersin scripts (\a,\c, \d, \o, \r,\t,\v, \x). These can cause similar problemswith scripts written for otherseds. 

-i clobbers read-only files

In short, ‘sed -i’ will let you delete the contents ofa read-only file, and in general the -ioption(seeInvocation) lets you clobberprotected files. This is not a bug, but rather a consequenceof how the Unix filesystem works.

The permissions on a file say what can happen to the datain that file, while the permissions on a directory say what canhappen to the list of files in that directory. ‘sed -i’will not ever open for writing a file that is already on disk. Rather, it will work on a temporary file that is finally renamedto the original name: if you rename or delete files, you're actuallymodifying the contents of the directory, so the operation depends onthe permissions of the directory, not of the file. For this samereason, sed does not let you use-i on a writeable filein a read-only directory, and will break hard or symbolic links when-i is used on such a file.

0a does not work (gives an error)

There is no line 0. 0 is a special address that is only used to treataddresses like 0,/RE/ as active when the script starts: ifyou write1,/abc/d and the first line includes the word ‘abc’,then that match would be ignored because address ranges must span at leasttwo lines (barring the end of the file); but what you probably wanted isto delete every line up to the first one including ‘abc’, and thisis obtained with0,/abc/d. 

[a-z] is case insensitive

You are encountering problems with locales. POSIX mandates that[a-z]uses the current locale's collation order – in C parlance, that means usingstrcoll(3) instead ofstrcmp(3). Some locales have acase-insensitive collation order, others don't.

Another problem is that [a-z] tries to use collation symbols. This only happens if you are on theGNUsystem, usingGNU libc's regular expression matcher instead of compiling theone supplied withGNU sed. In a Danish locale, for example,the regular expression^[a-z]$ matches the string ‘aa’,because this is a single collating symbol that comes after ‘a’and before ‘b’; ‘ll’ behaves similarly in Spanishlocales, or ‘ij’ in Dutch locales.

To work around these problems, which may cause bugs in shell scripts, setthe LC_COLLATE and LC_CTYPEenvironment variables to ‘C’.

s/.*// does not clear pattern space

This happens if your input stream includes invalid multibytesequences.posix mandates that such sequencesare not matched by ‘.’, so that ‘s/.*//’ will not clearpattern space as you would expect. In fact, there is no way to clearsed's buffers in the middle of the script in most multibyte locales(including UTF-8 locales). For this reason,GNU sed provides a `z'command (for `zap') as an extension.

To work around these problems, which may cause bugs in shell scripts, setthe LC_COLLATE and LC_CTYPEenvironment variables to ‘C’.

Next: Concept Index,Previous: Reporting Bugs,Up: Top

Appendix A Extended regular expressions

The only difference between basic and extended regular expressions is inthe behavior of a few characters: ‘?’, ‘+’, parentheses,and braces (‘{}’). While basic regular expressions requirethese to be escaped if you want them to behave as special characters,when using extended regular expressions you must escape them ifyou want themto match a literal character.

Examples:

abc?

becomes ‘abc\?’ when using extended regular expressions. It matchesthe literal string ‘abc?’.

c\+

becomes ‘c+’ when using extended regular expressions. It matchesone or more ‘c’s.

a\{3,\}

becomes ‘a{3,}’ when using extended regular expressions. It matchesthree or more ‘a’s.

\(abc\)\{2,3\}

becomes ‘(abc){2,3}’ when using extended regular expressions. Itmatches either ‘abcabc’ or ‘abcabcabc’.

\(abc*\)\1

becomes ‘(abc*)\1’ when using extended regular expressions. Backreferences must still be escaped when using extended regularexpressions.

Next: Command and Option Index,Previous: Extended regexps,Up: Top

Concept Index

This is a general index of all issues discussed in this manual, with theexception of thesed commands and command-line options.

• 0 address:Reporting Bugs
• Additional reading aboutsed: Other Resources
• addr1,+N:Addresses
• addr1,~N:Addresses
• Address, as a regular expression:Addresses
• Address, last line: Addresses
• Address, numeric: Addresses
• Addresses, insed scripts: Addresses
• Append hold space to pattern space:Other Commands
• Append next input line to pattern space:Other Commands
• Append pattern space to hold space:Other Commands
• Appending text after a line:Other Commands
• Backreferences, in regular expressions:The "s" Command
• Branch to a label, ifs/// failed: Extended Commands
• Branch to a label, ifs/// succeeded: Programming Commands
• Branch to a label, unconditionally:Programming Commands
• Buffer spaces, pattern and hold:Execution Cycle
• Bugs, reporting: Reporting Bugs
• Case-insensitive matching:The "s" Command
• Caveat — #n on first line:Common Commands
• Command groups: Common Commands
• Comments, in scripts: Common Commands
• Conditional branch: Extended Commands
• Conditional branch: Programming Commands
• Copy hold space into pattern space:Other Commands
• Copy pattern space into hold space:Other Commands
• Delete first line from pattern space:Other Commands
• Disabling autoprint, from command line:Invoking sed
• empty regular expression: Addresses
• Emptying pattern space: Reporting Bugs
• Emptying pattern space: Extended Commands
• Evaluate Bourne-shell commands:Extended Commands
• Evaluate Bourne-shell commands, after substitution:The "s" Command
• Exchange hold space with pattern space:Other Commands
• Excluding lines: Addresses
• Extended regular expressions, choosing:Invoking sed
• Extended regular expressions, syntax:Extended regexps
• File name, printing: Extended Commands
• Files to be processed as input:Invoking sed
• Flow of control in scripts:Programming Commands
• Global substitution: The "s" Command
• GNU extensions,/dev/stderr file: Other Commands
• GNU extensions,/dev/stderr file: The "s" Command
• GNU extensions,/dev/stdin file: Extended Commands
• GNU extensions,/dev/stdin file: Other Commands
• GNU extensions,/dev/stdout file: Footnotes
• GNU extensions,/dev/stdout file: Other Commands
• GNU extensions,/dev/stdout file: The "s" Command
• GNU extensions,0 address: Reporting Bugs
• GNU extensions,0 address: Addresses
• GNU extensions, 0,addr2 addressing:Addresses
• GNU extensions,addr1,+N addressing: Addresses
• GNU extensions,addr1,~N addressing: Addresses
• GNU extensions, branch ifs/// failed: Extended Commands
• GNU extensions, case modifiers ins commands: The "s" Command
• GNU extensions, checking for their presence:Extended Commands
• GNU extensions, disabling:Invoking sed
• GNU extensions, emptying pattern space:Reporting Bugs
• GNU extensions, emptying pattern space:Extended Commands
• GNU extensions, evaluating Bourne-shell commands:Extended Commands
• GNU extensions, evaluating Bourne-shell commands:The "s" Command
• GNU extensions, extended regular expressions:Invoking sed
• GNU extensions,g and number modifier interaction in s command:The "s" Command
• GNU extensions,I modifier: The "s" Command
• GNU extensions,I modifier: Addresses
• GNU extensions, in-place editing:Reporting Bugs
• GNU extensions, in-place editing:Invoking sed
• GNU extensions,L command: Extended Commands
• GNU extensions,M modifier: The "s" Command
• GNU extensions, modifiers and the empty regular expression:Addresses
• GNU extensions, ‘n~m’ addresses:Addresses
• GNU extensions, quitting silently:Extended Commands
• GNU extensions,R command: Extended Commands
• GNU extensions, reading a file a line at a time:Extended Commands
• GNU extensions, reformatting paragraphs:Extended Commands
• GNU extensions, returning an exit code:Extended Commands
• GNU extensions, returning an exit code:Common Commands
• GNU extensions, setting line length:Other Commands
• GNU extensions, special escapes:Reporting Bugs
• GNU extensions, special escapes:Escapes
• GNU extensions, special two-address forms:Addresses
• GNU extensions, subprocesses:Extended Commands
• GNU extensions, subprocesses:The "s" Command
• GNU extensions, to basic regular expressions:Reporting Bugs
• GNU extensions, to basic regular expressions:Regular Expressions
• GNU extensions, two addresses supported by most commands:Other Commands
• GNU extensions, unlimited line length:Limitations
• GNU extensions, writing first line to a file:Extended Commands
• Goto, in scripts: Programming Commands
• Greedy regular expression matching:Regular Expressions
• Grouping commands: Common Commands
• Hold space, appending from pattern space:Other Commands
• Hold space, appending to pattern space:Other Commands
• Hold space, copy into pattern space:Other Commands
• Hold space, copying pattern space into:Other Commands
• Hold space, definition: Execution Cycle
• Hold space, exchange with pattern space:Other Commands
• In-place editing: Reporting Bugs
• In-place editing, activating:Invoking sed
• In-place editing, Perl-style backup file names:Invoking sed
• Inserting text before a line:Other Commands
• Labels, in scripts: Programming Commands
• Last line, selecting: Addresses
• Line length, setting: Other Commands
• Line length, setting: Invoking sed
• Line number, printing: Other Commands
• Line selection: Addresses
• Line, selecting by number:Addresses
• Line, selecting by regular expression match:Addresses
• Line, selecting last: Addresses
• List pattern space: Other Commands
• Mixingg and number modifiers in the s command:The "s" Command
• Next input line, append to pattern space:Other Commands
• Next input line, replace pattern space with:Common Commands
• Non-bugs,0 address: Reporting Bugs
• Non-bugs, in-place editing:Reporting Bugs
• Non-bugs, localization-related:Reporting Bugs
• Non-bugs,N command on the last line: Reporting Bugs
• Non-bugs, regex syntax clashes:Reporting Bugs
• Parenthesized substrings: The "s" Command
• Pattern space, definition:Execution Cycle
• Perl-style regular expressions, multiline:Addresses
• Portability, comments: Common Commands
• Portability, line length limitations:Limitations
• Portability,N command on the last line: Reporting Bugs
• POSIXLY_CORRECT behavior, bracket expressions:Regular Expressions
• POSIXLY_CORRECT behavior, enabling:Invoking sed
• POSIXLY_CORRECT behavior, escapes:Escapes
• POSIXLY_CORRECT behavior,N command: Reporting Bugs
• Print first line from pattern space:Other Commands
• Printing file name: Extended Commands
• Printing line number: Other Commands
• Printing text unambiguously:Other Commands
• Quitting: Extended Commands
• Quitting: Common Commands
• Range of lines: Addresses
• Range with start address of zero:Addresses
• Read next input line: Common Commands
• Read text from a file: Extended Commands
• Read text from a file: Other Commands
• Reformat pattern space: Extended Commands
• Reformatting paragraphs: Extended Commands
• Replace hold space with copy of pattern space:Other Commands
• Replace pattern space with copy of hold space:Other Commands
• Replacing all text matching regexp in a line:The "s" Command
• Replacing onlynth match of regexp in a line: The "s" Command
• Replacing selected lines with other text:Other Commands
• Requiring GNUsed: Extended Commands
• Script structure: sed Programs
• Script, from a file: Invoking sed
• Script, from command line:Invoking sed
• sed program structure:sed Programs
• Selecting lines to process:Addresses
• Selecting non-matching lines:Addresses
• Several lines, selecting:Addresses
• Slash character, in regular expressions:Addresses
• Spaces, pattern and hold:Execution Cycle
• Special addressing forms: Addresses
• Standard input, processing as input:Invoking sed
• Stream editor: Introduction
• Subprocesses: Extended Commands
• Subprocesses: The "s" Command
• Substitution of text, options:The "s" Command
• Text, appending: Other Commands
• Text, deleting: Common Commands
• Text, insertion: Other Commands
• Text, printing: Common Commands
• Text, printing after substitution:The "s" Command
• Text, writing to a file after substitution:The "s" Command
• Transliteration: Other Commands
• Unbuffered I/O, choosing:Invoking sed
• Usage summary, printing: Invoking sed
• Version, printing: Invoking sed
• Working on separate files: Invoking sed
• Write first line to a file:Extended Commands
• Write to a file: Other Commands
• Zero, as range start address:Addresses

Previous: Concept Index,Up: Top

Command and Option Index

This is an alphabetical list of all sed commands and command-lineoptions.

• # (comments):Common Commands
• --binary: Invoking sed
• --expression: Invoking sed
• --file: Invoking sed
• --follow-symlinks:Invoking sed
• --help: Invoking sed
• --in-place: Invoking sed
• --line-length:Invoking sed
• --posix: Invoking sed
• --quiet: Invoking sed
• --regexp-extended:Invoking sed
• --separate: Invoking sed
• --silent: Invoking sed
• --unbuffered: Invoking sed
• --version: Invoking sed
• -b: Invoking sed
• -e: Invoking sed
• -f: Invoking sed
• -i: Invoking sed
• -l: Invoking sed
• -n: Invoking sed
• -n, forcing from within a script:Common Commands
• -r: Invoking sed
• -s: Invoking sed
• -u: Invoking sed
• : (label) command:Programming Commands
• = (print line number) command:Other Commands
• a (append text lines) command:Other Commands
• b (branch) command:Programming Commands
• c (change to text lines) command:Other Commands
• D (delete first line) command:Other Commands
• d (delete) command:Common Commands
• e (evaluate) command:Extended Commands
• F (File name) command:Extended Commands
• G (appending Get) command:Other Commands
• g (get) command:Other Commands
• H (append Hold) command:Other Commands
• h (hold) command:Other Commands
• i (insert text lines) command:Other Commands
• L (fLow paragraphs) command:Extended Commands
• l (list unambiguously) command:Other Commands
• N (append Next line) command:Other Commands
• n (next-line) command:Common Commands
• P (print first line) command:Other Commands
• p (print) command:Common Commands
• q (quit) command:Common Commands
• Q (silent Quit) command:Extended Commands
• r (read file) command:Other Commands
• R (read line) command:Extended Commands
• s command, option flags:The "s" Command
• T (test and branch if failed) command:Extended Commands
• t (test and branch if successful) command:Programming Commands
• v (version) command:Extended Commands
• w (write file) command:Other Commands
• W (write first line) command:Extended Commands
• x (eXchange) command:Other Commands
• y (transliterate) command:Other Commands
• z (Zap) command:Extended Commands
• {} command grouping:Common Commands

Table of Contents

• sed, a stream editor
• 1 Introduction
• 2 Invocation
• 3 sed Programs
• 3.1 How sed Works
• 3.2 Selecting lines with sed
• 3.3 Overview of Regular Expression Syntax
• 3.4 Often-Used Commands
• 3.5 The s Command
• 3.6 Less Frequently-Used Commands
• 3.7 Commands for sed gurus
• 3.8 Commands Specific to GNU sed
• 3.9 GNU Extensions for Escapes in Regular Expressions
• 4 Some Sample Scripts
• 4.1 Centering Lines
• 4.2 Increment a Number
• 4.3 Rename Files to Lower Case
• 4.4 Print bash Environment
• 4.5 Reverse Characters of Lines
• 4.6 Reverse Lines of Files
• 4.7 Numbering Lines
• 4.8 Numbering Non-blank Lines
• 4.9 Counting Characters
• 4.10 Counting Words
• 4.11 Counting Lines
• 4.12 Printing the First Lines
• 4.13 Printing the Last Lines
• 4.14 Make Duplicate Lines Unique
• 4.15 Print Duplicated Lines of Input
• 4.16 Remove All Duplicated Lines
• 4.17 Squeezing Blank Lines
• 5 GNU sed's Limitations and Non-limitations
• 6 Other Resources for Learning Aboutsed
• 7 Reporting Bugs
• Appendix A Extended regular expressions
• Concept Index
• Command and Option Index

Footnotes

[1] This applies to commands such as=,a, c, i, l, p. You canstill write to the standard output by using the worWcommands together with the /dev/stdoutspecial file

[2] Note thatGNU sed creates the backupfile whether or not any output is actually changed.

[3] Actually,ifsed prints a line without the terminating newline, it willnevertheless print the missing newline as soon as more text is sent tothe same output stream, which gives the “least expected surprise”even though it does not make commands like ‘sed -n p’ exactlyidentical tocat.

[4] This is equivalent top unless the -ioption is being used.

[5] This is equivalent top unless the -ioption is being used.

[6] Allthe escapes introduced here areGNUextensions, with the exception of \n. In basic regularexpression mode, settingPOSIXLY_CORRECT disables them insidebracket expressions.

[7] sed guru GregUbben wrote an implementation of thedc rpn calculator! It is distributed together with sed.

[8] This requiresanother script to pad the output of banner; for example

     #! /bin/sh

    

     banner -w $1 $2 $3 $4 |

       sed -e :a -e '/^.\{0,'$1'\}$/ { s/$/ /; ba; }' |

       ~/sedscripts/reverseline.sed

[9] Some implementationshave a limit of 199 commands per script

[10] which is the actual “bug” that prompted thechange in behavior