Now this is a real FAQ.
"How can I edit a file in-place using sed/awk/perl/whatever?"
or:
"I know that using >> I can append text to a file. How do I prepend text to a file?"
What these people usually mean is:
"How do I change/edit a file without having to create a temporary file?" (for some unknown reason)
Let's try to see what "in-place editing" really means, and why using temporary files (implicitly or explicitly) is the only way to do that reliably. Here we will limit our analysis to a pure shell environment using the commonly used filters, or stream-editing tools (sed, perl, awk and similar), which is a relatively common situation. Things start to change if we allow programming languages or special tools.
Note: almost every operating system, even when using "unbuffered" functions, maintains a low-level disk cache or buffer, so even when data is written to a file, it may not hit the disk immediately; similarly, when data is read, it may be coming from the buffer rather than disk. While this low-level OS caching is certainly something to be aware of, for the purposes of the following discussion it is entirely transparent and irrelevant to the points made, so it will be ignored here.
In-place?
Strictly speaking, "in-place" would really mean that: literally editing the very same file (the same inode). This can be done in principle, but:
- The program or utility must be designed to do that, meaning that it should arrange for the event that the file size increases, shrinks, or stays the same. Also, it must arrange things so data that hasn't yet been read is never overwritten. None of the usual text processing tools or filters is designed for this;
- It's dangerous: if something goes wrong in the middle of the edit (crash, disk full, etc.), the file is left in an inconsistent state.
None of the usual tools or editors do this; even when they seem to do so, they actually create a temporary file behind the scenes. Let's look at what sed and perl (two tools which are often said to be able to do "in-place" editing) do when the option -i is used.
sed
Sed has the -i switch for "in-place" editing. First of all, only some implementations of sed (GNU sed and BSD sed AFAIK) support -i. It's a nonstandard extension, and as such not universally available.
According to the documentation (at least GNU sed's), what sed does when -i is specified is create a temporary file, send output to that file, and at the end, that file is renamed to the original name. This can be verified with strace; even without strace, a simple
If you do use
Perl
Perl, similar to sed, has a -i switch to edit "in-place". And like sed, it creates a temporary file. However, the way Perl creates the temporary file is different. Perl opens and immediately unlink()s the original file, then opens a new file with the same name (new file descriptor and inode), and sends output to this second file; at the end, the old file is closed and thus deleted because it was unlinked, and what's left is a changed file with the same name as the original. This is more dangerous than sed, because if the process is interrupted halfway, the original file is lost (whereas in sed it would still be available, even if no backup extension was specified). Thus, it's even more important to supply a backup extension to Perl's -i, which results in the original file being rename()d rather than unlink()ed.
Another false in-place
By the way, here is a solution which is often described as "in-place" editing:
$ { rm file; command > file; } < file
("command" is a generic command that edits the file, typically a filter or a stream editor)
This works because, well, it's cheating. It really involves two files: the outer file is not really deleted by the rm command, as it's still open by virtue of the outer input redirection. The inner output redirection then really writes to a different disk file, although the operating system allows you to use the same file name because it's no longer "officially" in use at that point. When the whole thing completes, the original file (which was surviving anonymously for the duration of the processing, feeding command's standard input) is finally deleted from disk.
So, this kludge still needs the same additional disk space you'd need if you used a temporary file (ie, roughly the size of the original file). It basically replicates what Perl does with -i when no backup extension is supplied, including keeping the original file in the risky "open-but-deleted" state for the duration of the operation. So, if one must use this method, at least they should do
$ { mv file file.bak; command > file; } < file
But then, doing this is hardly different from using an explicit temporary file, so why not do that? And so...
Using an explicit temporary file
So, generally speaking, to accomplish almost any editing task on a file, temporary files should be used. Sure, if the file is big, creating a temporary file becomes more and more inefficient, and requires that an amount of available free space roughly the same size of the original file is available. Nonetheless, it's by far the only right and sane way to do the job. Modern machines should have no disk space problems.
The general method to edit a file, assuming command is the command that edits the file, is something along these lines:
$ command file > tempfile && mv tempfile file # or, depending on how "command" reads its input $ command < file > tempfile && mv tempfile file
To prepend data to a file, similarly do:
$ { command; cat file; } > tempfile && mv tempfile file
where command is the command that produces the output that should be prepended to the file.
The use of "&&" ensures that the original file is overwritten only if no errors occurred during the processing. That is to safeguard the original data in case something goes wrong. If preserving the original inode number (and thus permissions and other metadata) is a concern, there are various ways, here are two:
$ command file > tempfile && cat tempfile > file && rm tempfile # or $ cp file tempfile && command tempfile > file && rm tempfile
These commands are slightly less efficient than the previous methods, as they do two passes over the files (adding the cat in the first method and the cp in the second). In most cases, the general method works just fine and you don't need these latter methods. If you're concerned about the excruciating details of these operations, this page on pixelbeat lists many more methods to replace a file using temporary files, both preserving and not preserving the metadata, with a description of the pros and cons of each.
In any case, for our purposes the important thing to remember of these methods is that the old file stays around (whether under its original name or a different one) until the new one has been completely written, so errors can be detected and the old file rolled back. This makes them the preferred method for changing a file safely.
Sponges and other tricks
There are alternatives to the explicit temporary file, however they are somewhat inferior in the writer's opinion. On the upside, they have the advantage of (generally) preserving the inode and other file metadata.
One such tool is sponge, from the moreutils package. Its use is very simple:
command file | sponge file
As the man page says, what sponge does is "reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows for constructing pipelines that read from and write to the same file".
So, sponge accumulates output coming from command (in memory or, when it grows too much, guess where? in a temporary file), and does not open file again for writing until it has received EOF on input. When the incoming stream is over, it opens file for writing and writes the new data into it (if it had to use a temp file, it just rename()s that to file which is more efficient, although this results in changing the file's inode).
A barebone implementation of a sponge-like program in Perl would be like
#!/usr/bin/perl
# sponge.pl
while(<STDIN>) {
push @a, $_;
}
# EOF here
open(OUT, ">", $ARGV[0]) or die "Error opening $ARGV[0]: $!";
print OUT @a;
close(OUT);
This keeps everything in memory; it can be extended to use a temporary file (and, for that matter, it can likely be extended to also perform whatever job the filter that feeds its input does, but then we are leaving the domain of this article).
With this, one could do
command file | sponge.pl file
A similar functionality can be implemented using awk:
# sponge.awk
BEGIN {
outfile = ARGV[1]
ARGC--
}
{ a[NR] = $0 }
END {
for(i=1;i<=NR;i++)
print a[i] > outfile
}
These methods work, and they do edit the same file (inode), however they have the disadvantage that if the amount of data is huge, there is a moderately long period of time (while data is being written back to the file) during which part of the data is only in memory, so if the system crashes it will be lost.
The good old ed
If the editing to be done is not too complex, another alternative is the good old ed editor. A peculiarity of ed is that it reads its editing commands, rather than the data, from standard input. For example, to prepend "XXX" to each line in the file, It can be used as follows:
printf '%s\n' ',s/^/XXX/' w q | ed -s file
(the -s switch is to prevent ed from printing information on how many bytes it read/wrote; there's no harm in omitting it)
At least in most implementations, ed does create a temporary file, which it uses as support for the editing operations; when it is asked to save the changes, it writes them back to the original file. This way of working is mandated by the POSIX standard, that says that
The ed utility shall operate on a copy of the file it is editing; changes made to the copy shall have no effect on the file until a w (write) command is given. The copy of the text is called the buffer.
So, it should be clear that ed presents the same shortcomings of the sponge-like methods; in particular, when it's requested to perform a write (the "w" command), ed truncates the original file and writes the contents of the buffer into it. If the amount of data is huge, this means that there's a moderately long time window during which the file is in an inconsistent state, until ed has written back the whole data (and no other copy of the original data exists). Consider this if you're worried about unexpected things happening in the middle of the process.
"But I don't want to use a temp file!"
Ok. Having said all this, we still see that, for some mysterious reasons, people still try to do away with temporary files, and come up with "creative" solutions. Here are some of them. They are all broken and must not be used for any reason. "Kids, don't do this at home".
The pipe of death
People sometimes try this:
$ cat file | command > file # doesn't work
or also
$ command file | cat > file # doesn't work
Obviously none of these work, because the file is truncated by the shell as soon as the last part of the pipeline is run (for any practical purpose this means "immediately"). But, after thinking a bit about that, something "clicks" in the mind of whoever is writing the code, which generally leads to the following "clever" hack:
$ command file | { sleep 10; cat > file; } # DO NOT DO THIS
And that indeed appears to work. Except it's utterly wrong, and may bite you when you least expect it, with very bad consequences (things that seem to work are almost always much worse and dangerous than things that patently don't, because they can give a false sense of security). So, what's wrong with it?
The idea behind the hack is "let's sleep 10 seconds, so the command can read the whole file and do its job before the file is truncated and the fresh stuff coming from the pipe can be written to it". Let's ignore the fact that 10 seconds may or may not be appropriate (and the same goes for whatever value you choose to use). There's something much more seriously, fundamentally wrong there. Let's see what happens if the file is even moderately big. The right hand side of the pipeline will not consume any data coming from the pipe for 10 seconds (or however many seconds). This means that whatever command outputs, goes into the pipe and just sits there, at least until sleep is finished. But of course, a pipe cannot hold an infinite amount of data; rather, its size is usually fairly limited (like some tens of kilobytes, although it's implementation-dependent). Now, what happens if the output of command fills the pipe before sleep has finished? It happens that at some point a write() performed by command will block. If command is like most programs, that means that command itself will block. In particular, it will not read anything else from file. So it's entirely possible, especially if the input file is moderately large, and the output is accordingly large, that command will block without having read the input file fully. And it will remain blocked until sleep ends.
When that finally happens, there are at least two possible outcomes, depending on how exactly command reads its input and writes its output, the system's stdio buffering, the process scheduler, the shell and possibly some other factor (more on stdio buffering later).
If you're lucky (yes), you will end up writing a pipe's worth of output data into file and nothing more (of course losing its original contents, and the subsequent output that would have come from command). This is if you're lucky. Another, much worse, possibility is that command is unblocked when some of its output has already been written to file by the output redirection. What happens in that case is that the pipeline will enter an endless self-feeding loop, whereby cat writes the output of command to the file, but immediately after that command reads that same data again as its input, over and over. This causes file to grow without bounds as much as it can, possibly filling all the available space in the filesystem.
An alternative way of writing the same bad code, which probably makes the problem more evident is
$ cat file | { sleep 10; command > file; } # DO NOT DO THIS
Again, cat will block if file is big and the pipe is filled before the 10 seconds have passed.
It probably helps to state it more clearly: the above code has the potential to completely trash your system and render it unusable. Do NOT use it, for any reason. If you don't believe that and want to see it for yourself, try this on a filesystem that you can fill (a loopback filesystem is strongly suggested here):
# create a 100MB file
# dd if=/dev/zero of=loop.img bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 2.58083 s, 40.6 MB/s
# make a filesystem on it
# mke2fs loop.img
mke2fs 1.41.9 (22-Aug-2009)
loop.img is not a block special device.
Proceed anyway? (y,n) y
...
# mount it
# mount -o loop loop.img /mnt/temp
# cd /mnt/temp
# Just create a moderately big file
# seq 1 200000 > file
# Here we go
# sed 's/^/XXX/' file | { sleep 10; cat > file; } # DO NOT DO THIS
cat: write error: No space left on device
# ls -l
total 97611
-rw-r--r-- 1 root root 99549184 2010-04-02 20:08 file
drwx------ 2 root root 12288 2010-04-02 18:48 lost+found
# tail -n 3 file
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX8483
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX8484
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX#
# Uh-oh...
As the whole thing is completely nondeterministic, you might not get the same result (I had to repeatedly run it a few times myself and on different systems before getting the error). Nonetheless, you'll still have problems; if you don't enter the loop, then you'll end up with lots of missing data. Again: do NOT do the above for any reason. Imagine what could happen if this dangerous code is run as root by some cron job every night on an important server (hint: nothing good).
Buffers and descriptors
Another "solution" that is seen from time to time is something like
$ awk 'BEGIN{print "This is a prepended line"}{print}' file 1<>file # DO NOT DO THIS
This prevents the file from being truncated as it uses the <> notation which opens the file for reading and writing. So it would seem that this is the holy grail of in-place editing. But is it?
To understand why this "works" and why (you guessed it) it must not be used, let's approach the topic from a general point of view.
In general, during the editing or changing of the file, the overall amount of data that has to be written can be smaller, the same size, or larger than the data that it is supposed to replace. This poses problems.
Let's start with the case where the new data is the same size as the old, which is also the only one that can be made to work (although, again, it's not recommended). For example, with the code
awk '{gsub(/foo/, "bar"); print}'
the old data and the new data are all three characters; we also know that the old data is read before the new data is written out, so we may be able to do real "in-place" editing by doing something like
awk '{gsub(/foo/, "bar"); print}' file 1<>file # DO NOT DO THIS
this works because the "1<>file" syntax opens the file in read/write mode, and thus it's not truncated to zero length. Obviously, if the file is 1GB and the system crashes at some point in between, the data will be inconsistent. But it should be clear by now that we are already deep in the "don't do this" zone.
Let's see what happens if the replacement data is smaller than the original data.
$ cat file 100 200 300 400 500 $ sed 's/00/A/' file 1<>file $ cat file 1A 2A 3A 4A 5A 500
This is expected. Once the replacement data has been written back, what was in the original file past that point is left there, and sed (or other similar utilities) does not invoke truncate() or ftruncate(), because they are not designed to be used this way (and rightly so). So this can't work.
Now let's look at the most dangerous case: the changed data is longer than the original. It's the most dangerous because, unlike the previous one, sometimes it works and could lead people to mistakenly think that it can be safely done.
In theory, this shouldn't even work; after all, the first time a chunk of data that is bigger than the original data is written back, some data that has not been read yet will be overwritten, leading to data corruption at a minimum. However, there are some circumstances that may make it look as if it worked, although (did I say that already?) it shouldn't really be done as it's quite risky. The following analysis was performed on Linux and is thus specific to that system, but the concepts are general.
The first thing to observe is that, when using filters or streaming editors, reads happen before writes (obviously). So, say, the program might read 10 bytes, and write back 20 bytes, or so. This doesn't seem to help much, but the second thing to observe is that I/O operations are usually buffered; that is, most programs use buffered I/O calls, like fread() or fwrite(). These calls don't read and write directly to the file (as read() and write() would), but instead use some internal buffers (usually implemented by the C library) whose purpose is to "accumulate" data; when the application fread()s, say, 10 bytes, 4096 bytes are read instead and put in the read buffer (and 10 are returned to the application); when the application fwrite()s 20 bytes, these are written into an output buffer, and only when this buffer is full (again, perhaps 4096 bytes) it is written to the actual file. If the application's standard I/O descriptors are not connected to a terminal (and if the application does not call
We can confirm that this is indeed the case when, for example looking at the output of strace, we see that reads and writes happen in big chunks which do not correspond to the expected usage pattern of the application. For example, on this system the buffer size seems to be 4096 bytes. How does this matter for our problem? It matters because this buffering, specifically output buffering, is what makes the "write back more than was read" case work in some cases (but which is, instead, is a recipe for disaster).
So how does output buffering help? Let's go through the awk example we started from:
$ cat file
This is line1
This is line2
This is line3
This is line4
This is line5
$ awk 'BEGIN{print "This is a prepended line"}{print}' file 1<>file # DO NOT DO THIS
$ cat file
This is a prepended line
This is line1
This is line2
This is line3
This is line4
This is line5
This apparently miraculous outcome is possible because of I/O buffers. Let's have a look at the output of strace:
$ strace awk 'BEGIN{print "This is a prepended line"}{print}' file 1<>file
...
open("file", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=70, ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff555abe30) = -1 ENOTTY (Inappropriate ioctl for device)
fstat(3, {st_mode=S_IFREG|0644, st_size=70, ...}) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
read(3, "This is line1\nThis is line2\nThis"..., 70) = 70
read(3, "", 70) = 0
close(3) = 0
write(1, "This is a prepended line\nThis is li"..., 95) = 95
exit_group(0)
There's something strange there: why did print "This a prepended line"
Let's use ltrace, which can show library calls as well as system calls, to confirm our guesses (the output has been cleaned up in some places for clarity):
$ ltrace -S -n3 awk 'BEGIN{print "This is a prepended line"}{print}' file 1<>file
...
121 fileno(0x7f06e270b780) = 1
...
213 fwrite("This is a prepended line", 1, 24, 0x7f06e270b780 <unfinished ...>
214 SYS_fstat(1, 0x7fff663f4050) = 0
215 SYS_mmap(0, 4096, 3, 34, 0xffffffff) = 0x7f06e2daa000
216 <... fwrite resumed> ) = 24
217 __errno_location() = 0x7f06e2d9a6a8
218 fwrite("\n", 1, 1, 0x7f06e270b780) = 1
...
226 open("file", 0, 0666 <unfinished ...>
227 SYS_open("file", 0, 0666) = 3
228 <... open resumed> ) = 3
...
246 read(3, <unfinished ...>
247 SYS_read(3, "This is line1\nThis is line2\nThis"..., 70) = 70
248 <... read resumed> "This is line1\nThis is line2\nThis"..., 70) = 70
...
254 fwrite("This is line1", 1, 13, 0x7f06e270b780) = 13
255 __errno_location() = 0x7f06e2d9a6a8
256 fwrite("\n", 1, 1, 0x7f06e270b780) = 1
257 _setjmp(0x64d650, 0x1d64ceb, 0x1d635b0, 0, 0) = 0
258 __errno_location() = 0x7f06e2d9a6a8
259 fwrite("This is line2", 1, 13, 0x7f06e270b780) = 13
260 __errno_location() = 0x7f06e2d9a6a8
261 fwrite("\n", 1, 1, 0x7f06e270b780) = 1
262 _setjmp(0x64d650, 0x1d64cf9, 0x1d635b0, 0, 0) = 0
263 __errno_location() = 0x7f06e2d9a6a8
264 fwrite("This is line3", 1, 13, 0x7f06e270b780) = 13
265 __errno_location() = 0x7f06e2d9a6a8
266 fwrite("\n", 1, 1, 0x7f06e270b780) = 1
267 _setjmp(0x64d650, 0x1d64d07, 0x1d635b0, 0, 0) = 0
268 __errno_location() = 0x7f06e2d9a6a8
269 fwrite("This is line4", 1, 13, 0x7f06e270b780) = 13
270 __errno_location() = 0x7f06e2d9a6a8
271 fwrite("\n", 1, 1, 0x7f06e270b780) = 1
272 _setjmp(0x64d650, 0x1d64d15, 0x1d635b0, 0, 0) = 0
273 __errno_location() = 0x7f06e2d9a6a8
274 fwrite("This is line5", 1, 13, 0x7f06e270b780) = 13
275 __errno_location() = 0x7f06e2d9a6a8
276 fwrite("\n", 1, 1, 0x7f06e270b780) = 1
277 read(3, <unfinished ...>
278 SYS_read(3, "", 70) = 0
279 <... read resumed> "", 70) = 0
280 __errno_location() = 0x7f06e2d9a6a8
...
284 close(3 <unfinished ...>
285 SYS_close(3) = 0
286 <... close resumed> ) = 0
287 free(0x1d64cd0) = <void>
288 __errno_location() = 0x7f06e2d9a6a8
289 fflush(0x7f06e270b780 <unfinished ...>
290 SYS_write(1, "This is a prepended line\nThis is"..., 95) = 95
291 <... fflush resumed> ) = 0
292 fflush(0x7f06e270b860) = 0
293 exit(0 <unfinished ...>
294 SYS_exit_group(0 <no return ...>
295 +++ exited (status 0) +++
Awk uses buffered I/O (ie, fread()/fwrite()), and in line 121 the actual file descriptor corresponding to the object at address 0x7f06e270b860 (presumably a pointer to a FILE object for stdout) is obtained, which is 1 (ie, standard output).
Lines 213-218 are where the print statement in the BEGIN block is executed; note that no write system call is performed, so data is written to the C library buffer, not to the file. Lines 226-228 open the file for reading, as part of awk's normal processing before starting its loop, and lines 246-248 read the contents of the file (since input is buffered, the call to fread() triggers a read() system call that reads the whole file in the input buffer). Line 254 and following is where the main body of the awk program (ie, "{print}") is executed: again, all the data goes into the C library buffer, which already contained the line printed in the BEGIN block.
Line 284 closes the file descriptor used to read the file. Up to here, the file is still unchanged. Then at line 289, standard output is flushed, and only now data is written to the file (line 290).
So the output buffer effectively saves our bacon here. As a further test, let's run the command again but with output buffering disabled (using the neat stdbuf utility from GNU coreutils):
$ stdbuf -o0 awk 'BEGIN{print "This is a prepended line"}{print}' file 1<>file
# hangs, press ctrl-C
^C
$ ls -l file
-rw-r--r-- 1 waldner waldner 10298496 Jan 28 15:04 file
$ head -n 20 file
This is a prepended line
This is a prepended line
e2
This is line3
This is line4
This is line5
s is line4
This is line5
s is line4
This is line5
s is line4
This is line5
s is line4
This is line5
s is line4
This is line5
s is line4
This is line5
s is line4
This is line5
So this finally shows that (as expected) writing more than is read can't work, and it's only because of I/O buffering that it sometimes appears to work. And obviously, it's not known a priori whether I/O will be buffered (depends on the actual program code, and other things). Even if the POSIX standard requires that some functions use buffered I/O if they don't refer to "an interactive device", there's no guarantee that the application will use those functions (eg,
But if the above is not enough, let's continue this wicked game, and let's assume that we can rely on output buffering. Even in this case, we soon run into trouble.
Obviously, write buffering only provides a temporary storage for an amount of data that is less than or equal to the buffer size itself (eg, 4096 bytes). When the buffer is full, it is written out to the file. This means that the (already poor) protection provided by output buffering vanishes as soon as the size difference between read data and written data becomes greater than the buffer size. At that point, the output buffer is written to disk, and overwrites data that has not been read yet, thus disaster ensues again (data loss at a minimum, and potential endless loop with the file growing, depending on how the program exactly transforms the data). It's easy to verify; sticking to awk again,
# Let's prepend more than 4096 bytes to our file
$ awk 'BEGIN{for(i=1;i<=1100;i++)print i}1' file 1<>file
# after a while...
awk: write error: No space left on device
# Let's recreate the file
$ printf 'This is line 1\nThis is line 2\nThis is line 3\nThis is line 4\nThis is line 5\n' > file
# Let's try writing 5000 bytes at once now
$ awk 'BEGIN{printf "%05000d\n", 1}1' file 1<>file
$ cat file
[snip]
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
$ wc -l file
2 file
As can be seen above, whether the outcome is endless loop or "just" data corruption depends on how the program transforms the data.
A hall of shame
Now, knowing how it works, just for completeness, here is a little hall of shame that combines "ideas" from the bad techniques just described. It's provided to clearly state that these commands (and similar ones) must never ever be used.
$ sed 's/foo/longer/g' file 1<>file # DO NOT DO THIS
# prepend data to a file. Some smart cats detect this and complain
$ { command; cat file; } 1<>file # DO NOT DO THIS
# let's throw pipes into the mix
# prepend a file, bash
$ cat <(cat file1 file2) 1<>file2 # DO NOT DO THIS
# prepend a file, POSIX sh
$ cat file1 file2 | cat 1<>file2 # DO NOT DO THIS
# prepend text, "fooling" cat
$ { command; cat file; } | cat 1<>file # DO NOT DO THIS
Those using pipes are even more dangerous (if possible), as they introduce concurrency, which make the outcome even more unpredictable (process substitution in bash is implemented using a pipe, although it's not apparent from the above). Depending on how the processes are scheduled and where the data is buffered, the result can vary from success (unlikely), to self-feeding loop, to corrupted data. Again, try it for yourself a few times and you'll see. As an example, here's what happens with the last command above to prepend text to a file:
$ seq 100000 105000 > file
$ wc -l file
5001 file
$ { seq 1 2000; cat file; } | cat 1<>file
$ wc -l file
208229 file # should be 7001
$ seq 100000 105000 > file
$ wc -l file
5001 file
$ { seq 1 2000; cat file; } | cat 1<>file
$ wc -l file
194630 file # should be 7001
$ seq 100000 105000 > file
$ wc -l file
5001 file
# now let's add more data
$ { seq 1 20000; cat file; } | cat 1<>file
^C
$ ls -l file
-rw-r--r-- 1 waldner users 788046226 2010-05-09 15:26 file
# etc.
Conclusions
The bottom line of all this is that, to perform almost any editing/changing task on a file, you must use a temporary file, and for very good reasons. Also, it's much better if that file is explicit.
Update 28/12/2012:
It was brought to my attention that there is another way to write to the file without creating a temporary file. Before showing it, let me repeat that this is a bad idea, unless you REALLY know what you're doing (and even then, think many times about it before doing it).
So, at least with bash, the various expansions that the shell performs (variable expansion, command substitution, etc.) happen before redirections are set up; this makes sense, as one could do
mycommand > "$somefile"
so the variable $somefile needs to be expanded before the redirection can be set up. How can this be exploited for in-place editing (true in-place, in this case)? Simple, by dong this:
printf '%s\n' "$(sed 's/foo/bar/g' file)" > file # another one for the hall of shame
Of course, the output of the command substitution is temporarily stored in memory, so if the file is big, one may get errors like:
$ printf '%s\n' "$(sed 's/foo/bar/g' bigfile)" > bigfile -bash: xrealloc: ../bash/subst.c:658: cannot allocate 378889344 bytes (1308979200 bytes allocated) Connection to piggy closed.
Which, must be admitted, isn't as bad as some of the methods previously described because in this case, at least, the file isn't touched, that is, it's still as it was before running the command, rather than some intermediate inconsistent state.
Another, perhaps less obvious, problem with that approach is that (again, at least with bash) literal strings (such as the second argument to printf in the example) cannot contain ASCII NULs, so if the output of command substitution contains them, they will be missing in the result.
Update 2 23/02/2013:
For those who want real in-place editing, the Tie::File module of Perl is a way to do true in-place editing (same file, same inode) which also takes care of doing all the dirty work of expanding/shrinking the file. Basically, it presents the file as an array, and the code just has to modify the array; the changes are then converted to actual file changes on disk. Of course, all the caveats apply (file is inconsistent while it's being operated on) and, on top of that, performance will degrade as the file size (or amount of changes) increase. As they say, you can’t have your cake and eat it too.
Nevertheless, considering what it has to do, the Tie::File module is a really awesome piece of software.
As an example of a very basic usage, here are some simple operations (but there's no limit to the possibilities).
#!/usr/bin/perl use Tie::File; use warnings; use strict; my $filename = $ARGV[0]; my @array; tie @array, 'Tie::File', $filename or die "Cannot tie $filename"; $array[9] = 'newline10'; # change value of line 10 splice (@array, 0, 5); # removes first 5 lines for (@array) { s/foo/longerbar/g; # sed-like replacement } # etcetera; anything that can be done with an array can be done # (but see the CAVEATS section in the documentation) untie @array;
Sample run:
$ cat -n file.txt 1 this line will be deleted 1 2 this line will be deleted 2 3 this line will be deleted 3 4 this line will be deleted 4 5 this line will be deleted 5 6 this line will not be deleted foo 7 foo foo abc def 8 hello world 9 something normal 10 something weird 11 something foobar $ ls -i file.txt 11672298 file.txt $ ./tiefile_test.pl file.txt $ cat -n file.txt 1 this line will not be deleted longerbar 2 longerbar longerbar abc def 3 hello world 4 something normal 5 newline10 6 something longerbarbar $ ls -i file.txt 11672298 file.txt
Yes, it really is that simple. Now don't complain that it's slow or a crash messed up things.
Update 3 17/03/2015:
If (and only if)
- the replacement is exactly the same length of the part to be replaced
- you know exactly the position in the file where the replacement should be written
- you're felling brave
another possibility is the venerable dd program. The trick is that it's possible to tell dd to not delete the output file, using the
$ printf "newtext" | dd of=myfile seek=199 bs=1 conv=notrunc 7+0 records in 7+0 records out 7 bytes (7 B) copied, 3.4565e-05 s, 86.8 kB/s
and have the original file overwritten just where it needs to be. The reason for the "same length" requirement should hopefully be obvious.
No need to say that it's quite easy to screw up, but depending on the exact use case (eg binary editing), this might be a viable solution.
