Why does my ksh shell script die unexpectedly?

[ndedhia1]

I have a shell script that ssh over to another box, runs a cattail command on a log file, and redirects the output over a file on a different directory on the local box.
The name of the shell script is: desktop_SHM.ksh

#!/bin/ksh
 
ssh infrap@ics0001 '/sbt/prod/infra/bin/cattail /sbt/prod/infra/v2run_dir/log/alarmNotification.log' > /infra/sbt-support/supportTools/hsqldb/ics/tmp/alarmNo
tification.log.orig
 

I have this shell script running as a cronjob at 7:30am, monday to friday, but for some reason, when i come in at 8:00am and check to see if it is running, it has died at 7:30am after running once.

30 7 * * 1-5 /infra/sbt-support/supportTools/transferOutliers/bin/desktop_SHM.ksh > /dev/null 2>/infra/sbt-support/supportTools/logs/desktop_SHM.err
 

The desktop_SHM.err file is blank.

If I run this script from the command prompt, it works fine all day till I manually kill it, however if I run this script from the command prompt, but in the background, when i change directories it stops also.

[7]+  Stopped                 desktop_SHM.ksh
 

I am unsure why this is happening if someone can please help me.

[sicarie]

You're SSH'ing from local box A into a remote box B to redirect a file to a different location in remote box B?

Why not just set a cron on box B to copy the file every 5 minutes?

If it's redirecting that output back to box A, why not just send it over by utilizing syslog on box B and a collector on A? That would be close to live and not dependent on cron...

[ndedhia1]

That is basically correct.
If i run the script in the background, it works fine also as long as I dont change directories or run too many other commands on the command prompt. It seems as though even if i am running it in the background, I have to leave that session alone.

Running the script in the terminal manually definitely works fine alllll day long.

There is nothing of interest in the /var/log directory.
I believe the script exits after cattailing over the correct data 1 time. I think after the cronjob runs my script, it runs once, and then dies because whatever is running the crobjob, either dies or starts some other job, and my script dies because I feel as though its very dependent on the session that runs it. I am not sure if i am explaining this right, but if you have any more questions, please let me know.

thanks for the help!!

[sicarie]

Sorry, I wasn't paying attention to the way the solution was architected originally and revised my response. Any thoughts to the solutions presented? Or was this assigned and needs to be a modification of what is there right now?

[ndedhia1]

I need the data in the logs completely real-time. I thought about that solution, but even having it copy over every 1 min is too long of a lag time.
I believe this can be done the way that i am doing it manually, but I am not understanding why it is dying as a cronjob or dying periodically as a job that is run in the background.

[sicarie]

Have you looked into syslog? It seems that sending the individual event as soon as it comes across would be less intensive than ensuring an ssh'ing cronjob was active.

[ndedhia1]

i havent looked into that and definitely will. Thanks!

[ndedhia1]

the file that i am collecting data from is actually running syslog to capture its content. I need the captured contents also for an app that I am running.
Can I run syslog on a file and can you give me examples of how this works.

thanks

[ndedhia1]

is there anything you see in the shell script or the way I am running the cronjob that I am doing wrong?
Is there a ssh switch that I am missing or something that would allow for this to work?

[ndedhia1]

A lot of other threads that I have read, in which people are having this same problem, are saying that this is an environmental issue. Is this something I could be doing wrong, because I am not really using any variables?
I added these two lines to my shell script just to make sure that the file I am writing to has read/write access.
I am creating an empty file called alarmNotification.log.orig and then giving it 777 permission

>/infra/sbt-support/supportTools/hsqldb/ics/tmp/alarmNotification.log.orig
chmod 777 /infra/sbt-support/supportTools/hsqldb/ics/tmp/alarmNotification.log.orig
 

do you think that this could be the issue?
I wont be able to test it till tomorrow morning when the cronjob runs again.

[sicarie]

By 'environmental' I beleive they're saying the entire environment from your computer to server to switch to other server, etc... not just the environment variables in a shell script.

I really believe syslog would accomplish what you'd like to do, but it does take a bit to configure. You obviously have SSH keys set up, so I'd recommend putting the cron to every minute (or something reasonable, and check the behavior. Then try setting under 1 minute and look at the behavior.

Another option, outside syslog, is to set some sort of network share or mapped hard drive on both that you could mount locally on your computer. Then it would look like it was both logging locally and reading the file locally. That depends on what your'e doing though, especially if there are a lot of reads going across the network, or if you were manipulating the data.

I'd still recommend looking in /etc/syslog.conf to try to mimic the behavior (though yours right now isn't turned on, it'll give an outline of what it looks like), and then try to figure out how to set a listener up on your computer. One of the most popular and more up to date versions of syslog is called syslog-ng.

[ndedhia1]

Unfortunately,
I am not the network admin where I can configure many things on this box to allow syslog to work and also can not create a shared or mapped hard drive on both boxes.

I had this working in the past using Rexec, but last week, Rexec was disabled on the destination server so this no longer worked for me giving me a Connection refused error:

$ desktop_SHM.ksh.081611
server =ics0001,userName=infrap,password=infrap,cmd=/sbt/prod/infra/bin/cattail /sbt/prod/infra/v2run_dir/log/alarmNotification.log
 
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:367)
        at java.net.Socket.connect(Socket.java:529)
        at java.net.Socket.connect(Socket.java:477)
        at java.net.Socket.<init>(Socket.java:374)
        at java.net.Socket.<init>(Socket.java:187)
        at com.cboe.utility.RexecClient.<init>(RexecClient.java:76)
        at com.cboe.utility.RexecClient.<init>(RexecClient.java:17)
        at com.cboe.util.outlierMonitor.TransferOutliers.getOutliers(TransferOutliers.java:62)
        at com.cboe.util.outlierMonitor.TransferOutliers.main(TransferOutliers.java:168)
 

this is the shell script that was working for me up until last week:

#
 
java -DUniqueID=desktop_SHM \
     -DSERVER=ics0001 \
     -DREMOTEDIR=/sbt/prod/infra/v2run_dir/log/ \
     -DREMOTEFILE=alarmNotification.log \
     -DLOCALFILE=/infra/sbt-support/supportTools/hsqldb/ics/tmp/alarmNotification.log.orig \
     -DREMOTECOMMAND=/sbt/prod/infra/bin/cattail \
     -DUSERNAME=userID \
     -DPASSWORD=userPass \
     -classpath /infra/sbt-support/supportTools/transferOutliers/classes/Rexec.jar:/infra/sbt-support/supportTools/transferOutliers/classes/Utility.jar com.c
boe.util.outlierMonitor.TransferOutliers \
 

[sicarie]

Well, now you're getting into circumventing what someone else put in place, so I'm going to have to recommend you get whoever disabled that, your network admin, and your boss in a room to discuss your options (such as re-enabling, syslog, mapped network drive, or another idea).

It sounds like what you're doing is being blocked by your network/host admins, and I can't help you get around what they're putting in place as it violates the terms of use of this site.

Sorry and good luck!

[ndedhia1]

i dont want to get around what my network admins blocked. They wrote to me in an email that Rexec is being disabled and that i need to make this work using SSH.
I get what I need to do to work using SSH if i run the shell script via command line, but when i try to run it as a shell script, it dies after the first run. The file that is being appended to, alarmNotification.log.orig is updated up to 7:30am, which is when the cronjob is supposed to kick off. After that, it is never updated, which tells me that the cronjob died.
When i run it via command line, that file is updated all day long, every time a new log file appears in the alarmNotification.log file on the remote server.