LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 04-27-2024, 11:14 PM   #1
mfoley
Senior Member
 
Registered: Oct 2008
Location: Columbus, Ohio USA
Distribution: Slackware
Posts: 2,576

Rep: Reputation: 178Reputation: 178
how to escape $ (dollar sign) when echoing environment variable


I am looping through a list of files:
Code:
#!/bin/bash
cat fileList | while read
do
    ls "$REPLY"
done
The problem is that some of the files have '$' signs, e.g. "dir/$this file". echo, and printf attempt to expand $this as an environment variable giving "dir/ file" which of course the ls command doesn't find.

Is there a way to preserve the '$'? I probably need to change that to "dir/\$this file". How can I do that?
 
Old 04-27-2024, 11:30 PM   #2
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS, Manjaro
Posts: 5,678

Rep: Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712
Oh my.
There are a few things wrong with this, but let me ask before we start
IS this homework? Because HOW I answer depends upon WHY you are asking.

And while I wait for that information:
Do you understand how to use "read"? Because that is not how you use "read" in a loop".
And the related...
Where does the environment variable REPLY get set? Because it is never set in that script.

Where does the fileList come from? Has it structure, and what does it look like?

WHY in the world have you files containing shell special characters in the names???

Why call 'cat' in this script?

Is there some restriction that you must use BASH for this, because it may not be the optimal tool for the job.
 
Old 04-28-2024, 12:13 AM   #3
lvm_
Member
 
Registered: Jul 2020
Posts: 943

Rep: Reputation: 338Reputation: 338Reputation: 338Reputation: 338
It shouldn't. I tested your script and it prints dollars in filenames just fine on my machine. Weird. Maybe some shopt? Or maybe you simplified the bug out of it?

later: or is it bash at all? Could be busybox, if it's a router or something. What does 'ls -l /bin/bash' say?

Quote:
Originally Posted by wpeckham View Post
Do you understand how to use "read"? Because that is not how you use "read" in a loop".
And the related...
Where does the environment variable REPLY get set? Because it is never set in that script.
Ooooohhh.... this is the worst way to make a fool of yourself - a didactic rant, and you are wrong. This will smart :)

Last edited by lvm_; 04-28-2024 at 03:42 AM.
 
Old 04-28-2024, 07:05 AM   #4
michaelk
Moderator
 
Registered: Aug 2002
Posts: 25,749

Rep: Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928
The read command without a name variable will automatically save input to $REPLY.
Code:
read
echo $REPLY
The loop as written is technically correct syntax but not best practices. I agree that there must be something else going on because it should work. It might depend on the exact contents of your filelist.
Code:
filelist.txt
/dir1/$file1
/dir2/$file2

while read -r line
do
    echo "$line"
done < filelist.txt
 
Old 04-28-2024, 01:21 PM   #5
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS, Manjaro
Posts: 5,678

Rep: Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712
Thank you both for that correction. I had always used explicit variables and NEVER used the REPLY variable!

I would still like to see the answers to some of my other questions.

OF note, we both want to see the nature of that file. Without understand the input the processing and output are not determined.
 
Old 04-28-2024, 01:22 PM   #6
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS, Manjaro
Posts: 5,678

Rep: Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712
Quote:
Originally Posted by lvm_ View Post
later: or is it bash at all? Could be busybox, if it's a router or something. What does 'ls -l /bin/bash' say?
Note the first line (the #! line) of the script. To my knowledge, that would not work under busybox. (Although I have NOT tested that.)
 
Old 04-29-2024, 08:34 AM   #7
mfoley
Senior Member
 
Registered: Oct 2008
Location: Columbus, Ohio USA
Distribution: Slackware
Posts: 2,576

Original Poster
Rep: Reputation: 178Reputation: 178
OK, yes, I simplified the script too much. The problem is with echo. To clarify, fileList is ... a list of files! an example of an entry in this list containing a "$":
Code:
/mnt/public/Pension Files/1 Unpaid Leave-Military Leave/Military Leave reports/~$2023.02.25 Military Leave.xlsx
if I assign that to a variable:
Code:
$ x=$(echo "/mnt/public/Pension Files/1 Unpaid Leave-Military Leave/Military Leave reports/~$2023.02.25 Military Leave.xlsx")
$ echo $x
/mnt/public/Pension Files/1 Unpaid Leave-Military Leave/Military Leave reports/~023.02.25 Military Leave.xlsx
You can see that the "$2023" becomes "023". Why I am doing this in a while loop shouldn't matter to the question, and yes, I solved the problem another way, but short answer is that I have a list of files in a directory and subdirectories collected from another machine and I want to do something with these filepaths. The way I solved it for my purposes was to edit fileList and replace all $ with \$.

So, the real question is how to preserve '$' in an echo and not have echo attempt to replace with an env. variable.

To the other questions and comments:
Quote:
Originally Posted by wpeckham View Post
Oh my.
There are a few things wrong with this, but let me ask before we start
Do you understand how to use "read"? Because that is not how you use "read" in a loop".
Well, I've been using read in such loops for 20+ years. If you read input lines into a bash script some other way, I'd be interested to see that.
Quote:
And the related...
Where does the environment variable REPLY get set? Because it is never set in that script.
I believe this was answered by michaelk. In addition, using REPLY preserves leading whitespace in the line read whereas 'while read var' does not, even if you quote it inside the loop: echo "$var".
Quote:
Where does the fileList come from? Has it structure, and what does it look like?
I gave an acutal example above.
Quote:
WHY in the world have you files containing shell special characters in the names???
Not my choice. These are Windows filenames.
Quote:
Why call 'cat' in this script?
Why not? I could redirect stdin as shown in michaelk's example, but I think this is a matter of preference.
Quote:
Is there some restriction that you must use BASH for this, because it may not be the optimal tool for the job.
Sure, but I'd like to know if there is a way to deal with this in bash.
Quote:
Originally Posted by michaelk View Post
The read command without a name variable will automatically save input to $REPLY.
The loop as written is technically correct syntax but not best practices. ...
Hmmmm - what's not "best practice" about it?

Quote:
Originally Posted by wpeckham View Post
Note the first line (the #! line) of the script. To my knowledge, that would not work under busybox. (Although I have NOT tested that.)
This is pretty standard https://www.baeldung.com/linux/shebang:
Quote:
If we try to run a text file, execve expects the first two characters of the file to be “#!” (read “shebang” or “hashbang”) followed by a path to the interpreter that will be used to interpret the rest of the script.
Pretty much all my distro supplied bash scripts have "#!/bin/bash" or "#!/bin/sh" as the first line. And it can be used for scripts besides bash: "#!/usr/bin/perl" for .pl scripts.

So, if anyone has a solution to the echo "$var" problem where var contains '$', I'm still interested.

Last edited by mfoley; 05-02-2024 at 12:30 PM.
 
Old 04-29-2024, 09:30 AM   #8
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,930

Rep: Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321
x=$(echo "something") is just wrong. I mean it is valid, probably you wanted to do that, but in real life it is just x="something" (usually).
It is not the echo which removes the $, but the syntax you use means you want to evaluate variables within " ". That's why it is removed/evaluated. Actually there was no $2 (it is empty or undefined), therefore you got the result what you posted.

But anyway, we cannot show you any solution without knowing your script. At least the relevant part.

This cat | while is the typical UUoC (useless use of cat), you should avoid that. Post #4 offers a much better way.
 
Old 04-29-2024, 09:34 AM   #9
lvm_
Member
 
Registered: Jul 2020
Posts: 943

Rep: Reputation: 338Reputation: 338Reputation: 338Reputation: 338
It's not a problem where var contains '$', it's a quoting problem. To quote bash man page "Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, ...". Use single quotes or xargs.
 
Old 04-29-2024, 09:49 AM   #10
michaelk
Moderator
 
Registered: Aug 2002
Posts: 25,749

Rep: Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928Reputation: 5928
Quote:
Hmmmm - what's not "best practice" about it?
It falls under the useless use of cat.

I guess you need to sed to add an escape character...
 
Old 04-29-2024, 11:24 AM   #11
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS, Manjaro
Posts: 5,678

Rep: Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712Reputation: 2712
The use of cat generates an extra I/O call to load cat to do something for which it is not needed.
You can live without it and make your script faster and less vulnerable.

Allowing bash to interpret the string causes it to apply regex expansions, causing the problem. Echo, used as you did, is called as a bash internal. It is not needed and adds interpretation.

Escaping the character, as you did, is ONE way to avoid that problem. There are others. Not allowing the bash command line handling at your file name data would be optimal. Using a different tool than bash would be another.
 
Old 05-01-2024, 01:17 PM   #12
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 2,806

Rep: Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207
Quote:
using REPLY preserves leading whitespace in the line read whereas 'while read var' does not
Indeed! (I was surprised.)

I wouldn't bet this is true in other shells (Posix sh, ksh, zsh).
Portable is 'while IFS= read var'
The empty IFS environment variable makes read preserve leading whitespace. It is a temporary setting just for the read command (technically it happens between fork() and exec()).
 
1 members found this post helpful.
Old 05-02-2024, 12:42 PM   #13
mfoley
Senior Member
 
Registered: Oct 2008
Location: Columbus, Ohio USA
Distribution: Slackware
Posts: 2,576

Original Poster
Rep: Reputation: 178Reputation: 178
Quote:
Originally Posted by pan64 View Post
x=$(echo "something") is just wrong. I mean it is valid, probably you wanted to do that, but in real life it is just x="something" (usually).
It is not the echo which removes the $, but the syntax you use means you want to evaluate variables within " ". That's why it is removed/evaluated. Actually there was no $2 (it is empty or undefined), therefore you got the result what you posted.

But anyway, we cannot show you any solution without knowing your script. At least the relevant part.
Tough to show my actual script since I don't have it any more as I solved the problem another way. This is the best of my recollection:
Code:
cat fileList | while read
do
    x=$(echo "$REPLY" | sed 's#/mnt##')
    if [ -e "$x" ]; then echo yup; else echo nope; fi
done
I am trying remove or substitute characters in the $REPLY varliable. $x ends up with the "~$2023..." changed to "~023". Single quotes as lvm_ suggested can't be used because the $REPLY would not be substituted with its value.

And yes, I know I could use other tools. I'm interested specifically in a bash solution. If there isn't one, that's the answer.

Last edited by mfoley; 05-06-2024 at 05:17 PM.
 
Old 05-02-2024, 06:37 PM   #14
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 2,806

Rep: Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207Reputation: 1207
For reading file names use IFS= and -r
Use the built-in string operators (parameter modifiers).
Code:
while IFS= read -r fn
do
   x=${fn/\/mnt/} # deletes(substitutes) the first occurrence of /mnt (like sed 's#/mnt##')
#   x=${fn/#\/mnt/} # deletes(substitutes) a /mnt at the beginning (like sed 's#^/mnt##')
#   x=${fn#/mnt} # deletes a /mnt at the beginning (like sed 's#^/mnt##')
    if [ -e "$x" ]; then echo yup; else echo nope; fi
done < fileList
If you must use sed then this is almost safe:
Code:
x=$(echo "$fn" | sed 's#/mnt##')
Exception: $fn is a echo option like -n or -e
100% safe is printf with a format:
Code:
x=$(printf "%s\n" "$fn" | sed 's#/mnt##')
$-expressions within " " (like "$fn") are evaluated/substituted; no further substitution or expansion happens.
An assignment (like x=val) does not need quotes (x="val"). But a command does:
Code:
printf "%s\n" "$(printf "%s\n" "$fn" | sed 's#/mnt##')"
$( ) is a sub shell; the quotes in it don't interfere with the main shell.
For demonstration, you could write it as
Code:
printf "%s\n" "$(
# sub shell starts
printf "%s\n" "$fn" | sed 's#/mnt##'
# sub shell ends
)"

Last edited by MadeInGermany; 05-02-2024 at 07:16 PM.
 
2 members found this post helpful.
Old 05-03-2024, 12:40 AM   #15
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,930

Rep: Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321
Quote:
Originally Posted by mfoley View Post
I am trying remove or substitute characters in the $REPLY varliable. $x ends up with the "~@2023..." changed to "~023". Single quotes as lvm_ suggested can't be used because the $REPLY would not be substituted with its value.
I still think you brought this trouble on yourself. I mean it is all wrong. You did something wrong and are now trying to add some workarounds to make it work. The real solution would be to fix the initial issue, not to find a way to add patches.
But we would need to know how this variable was generated and if there was any way to make that differently.
Quote:
Originally Posted by mfoley View Post
And yes, I know I could use other tools. I'm interested specifically in a bash solution. If there isn't one, that's the answer.
In most cases it can be solved in bash, it has a huge amount of features. It is explained in the previous post. Sometimes it is not that efficient, and yes, this language is ugly, but there are many ways to make it work and also many ways to fail.
 
  


Reply

Tags
bash, escape



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
echoing a variable with a variable in its name? Speedy2k Linux - Newbie 5 12-31-2011 01:01 PM
[SOLVED] can't use ($) dollar sign in string ... php/fi 2.0.1 Linux Chips Programming 5 07-18-2011 09:37 AM
zsh PS1, dollar sign in the beginning, why? Mr. Alex Linux - Newbie 1 01-25-2011 09:48 AM
[SOLVED] Dollar sign in zsh Mr. Alex Linux - Newbie 3 01-24-2011 10:48 AM
Making grep find files containing '$' (the dollar sign char). stf92 Linux - Newbie 9 12-08-2010 11:16 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 03:06 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration