Problem running an Expect script within a Bash script

mbeipi · 03-14-2011, 07:21 PM

Hello, first of all let me state that I am not a systems administrator, I am just a humble geophysicist trying to retrieve some data from a server in an automated way.

I wrote an Expect script to do this:

Code:

#!/usr/bin/expect 
set year [lindex $argv 0]
set jday [lindex $argv 1]
spawn scp username@server:/somepath/$year/$jday/* ./.
expect "password:"
send "MyPassword\r"
interact

As you can see the data are in a directory structure that includes the year and the julian day.

Now, I know that it's better to do an RSA authentication or whatever it is and I shouldn't put the password in the script... I don't think the people who are giving me access to their data will let me do the RSA thing. So, let's assume that I have to use expect.

OK, the previous script works (only if I put that interact at the end, I have no idea why, if I don't put interact there then it gets to where the server is asking for the password and then returns to the prompt). The problem is that I don't want to sit there forever waiting for the data to download and then call the Expect script again with the next year jday combination. So I wrote a neat little bash script like so:

Code:

#!/bin/bash

while read year jday 
do

myscript.exp $year $jday

done <dates.file

where dates.file is something like:
2009 004
2009 087
...
you get the idea.

That does not work. It shows the server asking for the password and then I'm back at the command prompt.

Please help. I hope I do not anger you with my ignorance, I tried really hard to find an answer in other similar threads before breaking down and starting a new thread. TIA

Max

mbeipi · 03-14-2011, 07:27 PM

Update: I just tried removing the "interact" from the expect script, so now the bash script continues after the first year/julian daypair, gets to where it's asking for the password, and goes on to the next year/julianday pair and so on... without ever mgetting anything

jlinkels · 03-14-2011, 07:41 PM

I am very bad at debugging other's scripts, but:
put this near the beginning of you expect script:

Code:

exp_internal 1

and be surprised by the amount of useful information on your screen.
Also run you Bash script with sh -x yourscript to see what Bash internals are.

Somehow I have the idea that you should expect "password\r" or "password\r\n"
Once you have exp_internal 1 you'll see exactly what your server returns and how Expect matches that or not.

jlinkels

mbeipi · 03-15-2011, 12:22 AM

One more update:
I got rid of the "interact" and instead put in a "expect eof"

that KIND OF does the trick: it starts to perform the scp but then exits before it's done copying the files over!

I believe other people have run into this before and will look around tomorrow, but if anybody knows the answer it might save hours of time!

Thanks

BTW, thanks for the suggestion Jlinkles I tried that already, no luck!

grail · 03-15-2011, 02:11 AM

I am not all that familiar with expect scripts, but it would appear that as your scp is at least starting that the issue is it returns to the bash script and the next call to
the expect script, or at least to scp, is killing the first one. To this end I would look into finding out when the scp is finished prior to allowing
the bash loop to continue.
Whilst I am sure there are superior methods, something simple like the following may work:

Code:

wait $(pgrep -n scp)

This will wait for the most recent scp to conclude. You could additionally include this in an if and print an appropriate message should it not complete.

jlinkels · 03-15-2011, 02:21 AM

Just a calculated guess: put wait instead of interact.

This way Expect will wait until the child process is done. This is in line with your experience to wait for an end of file and grail's advice to wait until scp is done.

Besides you should wait after a spawn anyway in order to have Expect clean op child processes nicely when they do not terminate properly. But you see I often forget that myself.

jlinkels

grail · 03-15-2011, 02:30 AM

Thanks jinkels ... that sounds much better

mbeipi · 03-15-2011, 03:36 PM

Thanks guys, I appreciate your comments.
I gave

Code:

exp_internal 1

another shot after all and realized that my main problem was that expect was timing out, so I added this at the very beginning of my expect script:

Code:

set timeout -1

I then run scp in verbose mode and tell expect to...

Code:

expect "Exit status"

That works. It waits for the scp to complete and does not timeout. Using wait as you suggested may work too, but I think I may still have had the timeout issue. Thanks again for your suggestions!

PS I say "apparently solved" because it's running now and seems to be running well, it will be a long while before it actually finishes so I'll wait until then to declare victory.

colucix · 03-15-2011, 04:07 PM

Quote:

Originally Posted by mbeipi

Now, I know that it's better to do an RSA authentication or whatever it is and I shouldn't put the password in the script... I don't think the people who are giving me access to their data will let me do the RSA thing.

Just an aside note. As long as the remote server permits publickey authentication method, the process of creating the key pair and copying the public one onto the remote home directory is up to you. Nevertheless, you should have learned something useful about expect, today!

magic_007 · 02-10-2018, 05:00 AM

Quote:

Originally Posted by mbeipi

Thanks guys, I appreciate your comments.
I gave

Code:

exp_internal 1

another shot after all and realized that my main problem was that expect was timing out, so I added this at the very beginning of my expect script:

Code:

set timeout -1

I then run scp in verbose mode and tell expect to...

Code:

expect "Exit status"

That works. It waits for the scp to complete and does not timeout. Using wait as you suggested may work too, but I think I may still have had the timeout issue. Thanks again for your suggestions!

PS I say "apparently solved" because it's running now and seems to be running well, it will be a long while before it actually finishes so I'll wait until then to declare victory.

Thanks @mbeipi, this worked for me too.