Pass variable into sed program

[Michael F. Stemper]

I'm in the process of modifying some existing scripts to account for
the fact that the format of their input has changed. The following
piping of multiple sed commands did the trick in one place:

***@hostname$ order=42
***@hostname$ echo 7 | sed "s/,/) ($order,/g" | sed "s/^/($order,/"
| sed "s/$/)/"
(42,7)
***@hostname$ echo 2,3,5,7 | sed "s/,/) ($order,/g" | sed
"s/^/($order,/" | sed "s/$/)/"
(42,2) (42,3) (42,5) (42,7)
***@hostname$

Exactly what I needed. Then, I found another spot in the same script
where I needed the same idiom. I naturally decided to move these three
commands to a sed program:

***@hostname$ cat OldForm.sed
s/,/) ($order,/g
s/^/($order,/
s/$/)/
***@hostname$ echo 7 | sed -f OldForm.sed
($order,7)
***@hostname$

So far, so good.

Then, I tried to actually pass the variable in to the program as a
command-line argument:

***@hostname$ cat OldForm.sed
s/,/) ($1,/g
s/^/($1,/
s/$/)/
***@hostname$ echo 7 | sed -f OldForm.sed $order
sed: can't read 42: No such file or directory
***@hostname$

What's happened is obvious: sed is interpreting the "42" as the
name of a file to read, not as an argument.

I've gone through the O'Reilly book without finding out how to do
this. I've also spent some time searching on-line, and been told
over and over again how to use environment variables in one-liners
(like my opening example had).

What I have been unable to find is any description of how to pass
something into a sed *program*. This seems to be a very basic
action. Is it really true that it's not possible?

[Grant Taylor]

use environment variables in one-liners (like my opening example had).

I'd like to nitpick a moment. You aren't actually changing how sed
operates at all. Rather you're relying on the shell to dynamically
change the commands that you're asking sed to execute from the command
line. You aren't actually passing anything to sed. Further, sed isn't
doing anything different on the command line vs the script file.

What I have been unable to find is any description of how to pass
something into a sed *program*. This seems to be a very basic action. Is
it really true that it's not possible?

Try using the -e option to pass a command to populate the hold pattern
space with the variable. Then refactor your script file (program as you
called it) to g or G command to "copy / append hold space to pattern
space" in the proper locations in your sed script.

I've not tried this at all, so it may eat your dog. I don't even know
if you can combine -e and -f the way that I'm suggesting.

The only other thing that I can think of is to nest sed(s). Have the
outer execute the commands that the inner sed dynamically modifies and
prints to STDOUT. Or some such shenanigans.

This is also one of the reason that I like to use awk or m4 for some
text processing.

Good luck.

Please share your findings.

[Michael F. Stemper]

Post by Grant Taylor

use environment variables in one-liners (like my opening example had).

I'd like to nitpick a moment. You aren't actually changing how sed
operates at all. Rather you're relying on the shell to dynamically
change the commands that you're asking sed to execute from the command

I know that in the initial one-shot, the shell was doing the expanding.
It was only included to show potential respondents what I wanted to do.

Post by Grant Taylor

What I have been unable to find is any description of how to pass
something into a sed *program*. This seems to be a very basic action. Is
it really true that it's not possible?

Try using the -e option to pass a command to populate the hold pattern
space with the variable. Then refactor your script file (program as you
called it) to g or G command to "copy / append hold space to pattern
space" in the proper locations in your sed script.

Trying to get my head around hold and pattern spaces eventually
led me to:
<https://stackoverflow.com/questions/12833714/the-concept-of-hold-space-and-pattern-space-in-sed>
which says:

"[...] Hold spaces, pattern spaces, and 95% of the sed language [...]
became obsolete as soon as awk was invented in the mid-1970s [...] If
you are using more than s, g, and p (with -n) in sed then you are
almost certainly using the wrong tool."

The bit about it not having changed since the '70s probably explains
why it doesn't have the facility to accept arguments from the command
line, which seems like a no-brainer.

Post by Grant Taylor
This is also one of the reason that I like to use awk or m4 for some
text processing.

I took your and their advice and rewrote the entire script as a
python program.

Thanks for the pointer.

[Joe Rosevear]

Michael F. Stemper <***@gmail.com> wrote:

[snip]

Post by Michael F. Stemper
Then, I tried to actually pass the variable in to the program as a
s/,/) ($1,/g
s/^/($1,/
s/$/)/
sed: can't read 42: No such file or directory
What's happened is obvious: sed is interpreting the "42" as the
name of a file to read, not as an argument.
I've gone through the O'Reilly book without finding out how to do
this. I've also spent some time searching on-line, and been told
over and over again how to use environment variables in one-liners
(like my opening example had).
What I have been unable to find is any description of how to pass
something into a sed *program*. This seems to be a very basic
action. Is it really true that it's not possible?

Hi, I'm a little late in replying, but here is a solution.

I wrote a bash script called OldForm2.sh and I gave it execute
permission:

#!/bin/sh

sed "s/,/) ($1,/g" | \
sed 's/^/('$1',/' | \
sed 's/$/)/'

Here is a sample execution:

***@zillo:/tmp/joe/scratch$ order=42

***@zillo:/tmp/joe/scratch$ echo 2,3,5,7 | ./OldForm2.sh $order
(42,2) (42,3) (42,5) (42,7)

What I wrote is very close to what you wrote. The differences are:

1. It is a bash script instead of a sed file.

2. I pass $order not to sed, but to the bash script.

3. I use the continuation symbol "\" to arrange the script in a
format that resembles your sed file.

4. I was careful to use ' and " correctly. Note that for
demonstration I used " in the first line and ' in the second line.
The third line has no $1 so either ' or " could be used in the same
way.

Observe what I did in the second line. $1 is not *in* a string,
because it *is* a string. Bash concatenates strings placed end to
end. The use of " is different because it allows dereferencing of
positional parameters.

I hope this is not too late to be of value. By the way, thanks for the
interesting illustration of use of sed. Although I knew how to solve
the positional parameter problem, I did not know how to pass comma
separated values to sed as you did. Cool.

-Joe

[Michael F. Stemper]

Post by Joe Rosevear

Post by Michael F. Stemper
Then, I tried to actually pass the variable in to the program as a
s/,/) ($1,/g
s/^/($1,/
s/$/)/
sed: can't read 42: No such file or directory
What's happened is obvious: sed is interpreting the "42" as the
name of a file to read, not as an argument.
What I have been unable to find is any description of how to pass
something into a sed *program*. This seems to be a very basic
action. Is it really true that it's not possible?

Hi, I'm a little late in replying, but here is a solution.
I wrote a bash script called OldForm2.sh and I gave it execute
#!/bin/sh
sed "s/,/) ($1,/g" | \
sed 's/^/('$1',/' | \
sed 's/$/)/'
1. It is a bash script instead of a sed file.
2. I pass $order not to sed, but to the bash script.

Y'know, I usually do my sed-ing as one (or several) part(s) of a long
pipe. I have no idea why I was so bound and determined to make a sed
program this time. Especially since it was for a one-time data format
conversion.

Post by Joe Rosevear
I hope this is not too late to be of value.

By the time that you'd posted, I'd already worked around it. But, it
still helps to be reminded that there are multiple ways to skin any
cat.

Post by Joe Rosevear
By the way, thanks for the
interesting illustration of use of sed. Although I knew how to solve
the positional parameter problem, I did not know how to pass comma
separated values to sed as you did. Cool.

We both learned something then? Cool indeed.