need help doing a egrep and replace | Member | | Join Date: Jan 2009
Posts: 68
| |
Right now, I am using this line of code to get rid of data in a file called alarmNotification that I do not want: -
egrep "low debug.*\".*\"" $dbDir/alarmNotification.log
-
but I am having some problem. An entry like the one below in the alarmNotification.log file is getting deleted when I only want a part of the entry to be deleted, keeping the rest of the line in tact -
low debug 2009/3/9 8:30:21.01 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4600^A1236605421010^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY
- at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
-
at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
-
at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
-
at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
-
at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
-
at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
-
at java.lang.Thread.run(Thread.java:619),^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
-
I want this entry to look like this after I correct the problem: -
low debug 2009/3/9 8:30:21.01 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4600^A1236605421010^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
-
entries like these dont get effected at all by the egrep line: -
low debug 2009/3/9 8:30:18.05 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4589^A1236605418050^A23605^A23603^A2^Am[0]=23601^A60^AProdcas01v2cas0102^APOATP/CASQuote^A" 0
-
low debug 2009/3/9 8:30:19.52 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4590^A1236605419520^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:15:380 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(727). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=1171, delivered=570184, matched=0, discarded=742^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
-
Instead of an egrep, is there a global substitute or something that I could do.
thanks
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
I think u shuld change the regex for this.
As u use . also i thnk it is picking a wrong pattern.
Why cant u use like this
[a-z]a[z][a-z]\.[a-z]a-z]a-z]\. ==> for com.cboe
Raghu
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
So you mean have the regex like this: -
egrep "low debug.*\"[a-z]a[z][a-z]\.[a-z]a-z]a-z]\.*\"" $dbDir/alarmNotification.log
-
is the syntax correct: the a[z] and the a-z]?
thanks
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
This is the problem actually:
This regex: -
egrep "low debug.*\".*\"" $dbDir/alarmNotification.log
-
is looking for data between the two quotation marks:
When I hate data like this: -
low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4594^A1236605420470^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
-
at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
-
at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
-
at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
-
at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
-
at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
-
at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
-
at java.lang.Thread.run(Thread.java:619)
-
,^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
-
There are line break before each -
at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
-
and so there is no closing quote after this line: -
low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4594^A1236605420470^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
-
I want my output to have everything, including the "at com.cboe..." -
low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4593^A1236605420470^A15904^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99) at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467) at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388) at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109) at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111) at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27) at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203) at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146) at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51) at java.lang.Thread.run(Thread.java:619),^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A"
-
but because they are different lines, the egrep doesnt select them. What can I use that will egrep between multiple lines from the beginning to the end of the quotation marks.
This data below works below it searches between the quotation marks and this is all on one line: -
low debug 2009/3/9 8:30:19.53 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4592^A1236605419530^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:18:84 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(782). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=890, delivered=595531, matched=0, discarded=1542^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
-
thanks
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
Hi,
Is someone able to help me with this problem
thanks
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
Is there something that I can use to search in a file over more then 1 line.
I believe that egrep only works line by line but can i search between lines?
thanks
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
Hi,
Then i think you shuld wite your code using AWK.
I know AWK has this feature.
Raghu
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
Thanks gpraghuram,
Is there a code snippet you have that would help me
thanks
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
Hi,
I dont have a example code with me.
But check this link
and look for this "Multiline Records" in the page.
Raghu
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
I checked out the link and I understand what they are saying: - Multiline Records
-
-
By default, the input record separator RS recognizes a newline as the marker between records. As is the norm in awk, this can be changed to allow for multiline records. When RS is set to the null string, the newline character always acts as a field separator, in addition to whatever value FS might have.
-
But how do I change RS so that it looks between two quotation marks:
and the awk command that does that.
thanks
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
I used this snippet of code -
awk -Flow print "low debug.*\".*\"" $dbDir/alarmNotification.log
-
I wanted the lines to be separated by the word low like in these entries: - low debug 2009/3/9 8:30:19.53 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4591^A1236605419530^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:16:775 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(18). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=0, delivered=582743, matched=0, discarded=760^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
- low debug 2009/3/9 8:30:19.53 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4592^A1236605419530^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:18:84 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(782). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=890, delivered=595531, matched=0, discarded=1542^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
- low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4594^A1236605420470^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
-
at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
-
at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
-
at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
-
at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
-
at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
-
at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
-
at java.lang.Thread.run(Thread.java:619)
-
,^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
- low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4593^A1236605420470^A15904^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
-
at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
-
at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
-
at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
-
at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
-
at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
-
at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
-
at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
-
at java.lang.Thread.run(Thread.java:619)
-
,^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
-
and I wanted it to print everything in the line:
from the file: -
$dbDir/alarmNotification.log
-
BUT i get these errors: -
awk: syntax error near line 1
-
awk: bailing out near line 1
-
unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-17/alarmNotificationTmp.txt (No such file or directory)
-
What am i doing wrong?
thanks
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
Instead of command line write a awk program which will be easy. - BEGIN{
-
RS="";
-
FS="low"
-
}
-
{
-
/*write code here for processing the line*/
-
}
-
END{
-
}
Raghu
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
I created an AWK file called grepLogDebug.awk which has this code: -
BEGIN{
-
RS="";
-
FS="low"
-
}
-
{
-
egrep "low debug.*\".*\"" $dbDir/alarmNotification.log;
-
}
-
END{
-
and here is how I call the awk file: -
/infra/bin/gawk -f $workDir/grepLogDebug.awk
-
and i get this error: -
gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:6: egrep "low debug.*\".*\"" $dbDir/alarmNotification.log;
-
gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:6: ^ parse error
-
unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-18/alarmNotificationTmp.txt (No such file or directory)
-
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
I changed some of the code:
To call the AWK file: -
/infra/bin/gawk -f $workDir/grepLogDebug.awk dbDir=$dbDir
-
and in the AWK file: -
BEGIN{
-
RS="";
-
FS="low"
-
}
-
{
- printf "low debug.*\".*\"" > (dbDir "/alarmNotification.log");
-
}
-
END{
-
}
-
but when i run it,
it get stuck and doesnt finish creating the tables
thanks
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
But where u are giving file name .
call like this
/infra/bin/gawk -f $workDir/grepLogDebug.awk dbDir=$dbDir < $IP_FILE
also you cant directly get the value of dbdir inside the AWK script and there is a special syntax to pass input arguments to the AWK.
Check in gawk manual
Raghu
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
is the $IP_FILE the name of the file that I am trying to print from:
so for me, the $IP_FILE should be: $dbDir/alarmNotification.log so should my call looks like this? -
/infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log
-
and then inside my grepLogDebug.awk file
is my print line correct because I am still getting errors: -
BEGIN{
-
RS="";
-
FS="low"
-
}
-
{
-
printf "low debug.*\".*\"";
-
}
-
END{
-
thanks
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
printf is not ok.
printf in awk is similar to C syntax. - printf("low debug.*\".*\"");
Raghu
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
I have changed the printf statement and now I get these errors when i try to create the tables: -
unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-24/alarmNotificationTmp.txt (No such file or directory)
-
-
fatal: error reading input file `stdin': I/O error
-
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
Hi,
I think this discissuion is going on for a long time.
Why cant u post the code and the way u run your script and i can check it.
Raghu
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
Hello Raghu,
Here is the code that I am running: getAndArchiveIcsDbTest -
#!/bin/ksh
-
#/infra/sbt-support/supportTools/hsqldb/ics/setProperties
-
-
export javaDir=/usr/jdk/jdk1.5.0_06/bin
-
export icsHost=deepfrzd
-
export icsLogin=infra
-
export workDir=/infra/sbt-support/supportTools/hsqldb/ics
-
export tmpDir=/infra/sbt-support/supportTools/hsqldb/ics/tmp
-
export AppRoot=/infra/sbt-support/supportTools/hsqldb/ics/DBConnection
-
export hypDB=ICS_DB
-
export archiveDir=/infra/sbt-support/supportTools/hsqldb/ics/tmp
-
eval $(/infra/bin/EffectiveDate)
-
export dbDir="$tmpDir/$WebDate"
-
-
remoteDbDir="/infra/sbt-support/supportTools/hsqldb/ics/archive/$1"
-
remoteAlarmDir="/infra/sbt-support/supportTools/hsqldb/ics/archive/$1"
-
previousDatesDir="/infra/sbt-support/supportTools/hsqldb/ics/archive"
-
-
mkdir $dbDir
-
-
$workDir/getIcsDbFiles.ksh $remoteDbDir $remoteAlarmDir
-
- /infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
-
-
#egrep "low debug.*\".*\"" $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
-
-
$AppRoot/bin/runConvertMillis $dbDir
-
-
$AppRoot/bin/runDBConnectionHyp $AppRoot/etc/hypProperties $AppRoot/sql/createTextTables.sql $tmpDir/$WebDate/$hypDB
-
-
cd $tmpDir
-
tar -cf $WebDate.tar $WebDate
-
gzip -f $WebDate.tar
-
-
rm $WebDate.tar.gz
-
mv $WebDate $1
-
tar -cvf $1.tar $1
-
gzip $1.tar
-
mv $1.tar.gz $previousDatesDir
-
-
# mv -f $WebDate.tar.gz $archiveDir
-
# rm -rf $WebDate
-
# rmdir $WebDate
-
grepLogDebug.awk -
BEGIN{
-
RS="";
-
FS="low"
-
}
-
{
- printf("low debug.*\".*\"");
-
}
-
END{
-
}
-
here is a sample of the alarmNotification.log - low debug 2009/3/9 10:33:42.03 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7089^A1236612822030^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 10:33:38:211 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(2819). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=1970, delivered=41296526, matched=0, discarded=2453375^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
- low debug 2009/3/9 10:33:46.74 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7090^A1236612826740^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 10:33:44:486 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(16). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=1996, delivered=41322105, matched=0, discarded=2453391^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
- low debug 2009/3/9 10:33:46.86 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7091^A1236612826860^A19971^A19968^A3^Am[0]=19833^A160^AProdBC91x1HybridTradeServer1prdbc91b^AMarketDataQueue_11^Am[1]=19831^A160^AProdBC91x1HybridTradeServer1prdbc91b^AMarketDataQueue_11^A" 0
- low debug 2009/3/9 10:34:05.22 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7094^A1236612845220^A15601^A13503^A1^Am[0]=13501^Acom.cboe.infrastructureServices.persistenceService.UniqueConstraintException,No exception message given,Delegating stack trace to the causing exception.
-
java.sql.SQLException: ORA-00001: unique constraint (TRDSESS_10_OWNER.SBTORDERUK) violated
-
-
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
-
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
-
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
-
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
-
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
-
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)
-
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1167)
-
at oracle.jdbc.driver.OraclePreparedStatement.sendBatch(OraclePreparedStatement.java:3628)
-
at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:1119)
-
at com.objectwave.persist.RDBConnection.commit(RDBConnection.java:536)
-
at com.objectwave.persist.RDBBroker.commit(RDBBroker.java:132)
-
at com.objectwave.persist.BrokerChangeList.commitAll(BrokerChangeList.java:59)
-
at com.objectwave.persist.BrokerTransactionLog.commit(BrokerTransactionLog.java:88)
-
at com.objectwave.persist.BrokerTransactionLog.actualCommit(BrokerTransactionLog.java:53)
-
at com.objectwave.transactionalSupport.TransactionLog.commit(TransactionLog.java:298)
-
at com.objectwave.transactionalSupport.Session.commit(Session.java:51)
-
at com.cboe.infrastructureServices.foundationFramework.transactionManagement.BOSession.commit(BOSession.java:99)
-
at com.cboe.infrastructureServices.foundationFramework.utilities.Transaction.commit(Transaction.java:139)
-
at com.cboe.ohs.domain.OrdersQueueReader.processPersistOrderList(OrdersQueueReader.java:149)
-
at com.cboe.ohs.domain.OrdersQueueReader.addTransactionalBoundary(OrdersQueueReader.java:130)
-
at com.cboe.ohs.domain.OrdersQueueReader.dequeueAndaddTransactionBoundaries(OrdersQueueReader.java:63)
-
at com.cboe.ohs.domain.OrdersQueueReader.run(OrdersQueueReader.java:42)
-
at java.lang.Thread.run(Thread.java:619)
- ,^AProdBC10x1OHServerHybridprdbc10a^AOHServerHybrid^A" 0
-
I call the getAndArchiveIcsDbTest on the command line with a date as an argument..for example:
getAndArchiveIcsDbTest 2009-03-09
That is when i get the errors that I posted earlier.
If I uncomment out the egrep statement in the getAndArchiveIcsDbTest file and comment out the /infra/bin/gawk statement, it runs with no errors, but it does not print out the low debug statements that have multiple lines, from the alarmNotification.log file, which is what I am trying to fix.
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
I think problem is this line -
-
#/infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
-
#change the above line to this
-
-
/infra/bin/gawk -f $workDir/grepLogDebug.awk < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
-
Raghu
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
The code change got rid of the fatal error: -
fatal: error reading input file `stdin': I/O error
-
but I still get this exception error: -
unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-26/alarmNotificationTmp.txt (No such file or directory)
-
I do not get this error when I comment out the /infra/bin/gawk statement and uncomment the egrep statement.
Thanks
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
Another error that I found is that this line: -
/infra/bin/gawk -f $workDir/grepLogDebug.awk < $dbDir/alarmNotification.log
-
creates the output of this, which is incorrect: -
low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"
-
The line which I was using before: -
egrep "low debug.*\".*\"" $dbDir/alarmNotification.log
-
creates this output which is correct except for the issue I am trying to fix: -
low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A792^A1236577300600^A15806^A15402^A1^Am[0]=15401^A1^AProdfixcas08v2fix8^AALL^A" 0
-
low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A795^A1236577300600^A15404^A15402^A1^Am[0]=15401^A1^AProdfixcas08v2fix8^AALL^A" 0
-
low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A796^A1236577300600^A15806^A15402^A1^Am[0]=15401^A1^AProdfixcas15v2fix1b^AALL^A" 0
-
low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A797^A1236577300600^A15404^A15402^A1^Am[0]=15401^A1^AProdfixcas15v2fix1b^AALL^A" 0
-
low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A794^A1236577300600^A11329^A11327^A1^Am[0]=11326^A1^AProdfixcas08v2fix8^AALL^A" 0
-
low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A793^A1236577300600^A10704^A10702^A1^Am[0]=10701^A1^AProdfixcas15v2fix1b^AALL^A" 0
-
thanks
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
In the awk you are only combinig the multiple lines and u are not processing anything?
Also u have the replace logic still in egrep?
Raghu
| | Member | | Join Date: Jan 2009
Posts: 68
| | | re: need help doing a egrep and replace
are you talking about processing the data in the grepLogDebug.awk file?
i tried the egrep statement in the grepLogDebug.awk file: -
FS="low"
-
}
-
{
-
printf("low debug.*\".*\"");
- egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotification.log;
-
}
-
END{
-
}
-
and I call the statement like this in the getAndArchiveIcsDbTest file: -
/infra/bin/gawk -f $workDir/grepLogDebug.awk $WebDate < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
-
but I get this error: -
gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:7: egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotification.log;
-
gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:7: ^ parse error
-
unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-27/alarmNotificationTmp.txt (No such file or directory)
-
I am not sure where I am supposed to process anything and what I should do with the replace logic that is in the egrep. I though that my grepLogDebug.awk file would make it so that I would be searching over multiple lines and then the egrep: -
egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotification.log;
-
would be able to get the info that I need and sed commands: -
/infra/bin/gawk -f $workDir/grepLogDebug.awk $WebDate < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
-
would do the replacing and work that I need done on the data that is captured from the grepLogDebug.awk file.
I am trying to figure out where I am going wrong because I have the logic correct for searching and replacing if all my data is within the same line. I didnt think that because the data is scattered over multiple lines, I would have this much trouble.
| | Member | | Join Date: Mar 2007 Location: Manila,Philippines
Posts: 58
| | | re: need help doing a egrep and replace
hi all.,
it's been a long time since i replied/post here. while having some trouble right now doing my script using solaris, i'm getting error:
"grep -hblcnsviw" pattern error
the usual grep is not working here.i hope you could me.
thanks.
coaxfiber
| | Member | | Join Date: Mar 2007 Location: Manila,Philippines
Posts: 58
| | | re: need help doing a egrep and replace
coaxfiber> grep -E -e -hblcnsviw 'MTP3b' Health_1_Mar26_10:33
grep: illegal option -- E
grep: illegal option -- e
Usage: grep -hblcnsviw pattern file . . .
coaxfiber> grep -E -e 'MTP3b' -e 'Rnsap' Health_1_Mar26_10:33
grep: illegal option -- E
grep: illegal option -- e
Usage: grep -hblcnsviw pattern file . . .
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
HI,
Why cant u start a new thread instead of appending to an existing thread?
Raghu
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
Hi,
If u want to call egrep from awk then u shuld use system command.
If u want to replace something inside awk then u shuld use sub or gsub.
Better get a awk manual to read abt it.
Raghu
| | Member | | Join Date: Mar 2007 Location: Manila,Philippines
Posts: 58
| | | re: need help doing a egrep and replace
hi, can you help me out why i'm not allowed to post new thread.. thanks..
|  | Expert | | Join Date: Mar 2007 Location: Chennai
Posts: 1,266
| | | re: need help doing a egrep and replace
This shuld be answered by administrators.
Admins,
can u look into this issue.
Raghu
| | Member | | Join Date: Mar 2007 Location: Manila,Philippines
Posts: 58
| | | re: need help doing a egrep and replace
the contents of my scripts:
#!/usr/bin/ksh
integer x=1
while ((x<=20));
do
print $x;
if [ $x = 5 ]
then
x=10
fi
((x=x+1));
done
_____________
output:
1
2
3
4
5
11
12
13
14
15
16
17
18
19
20
-------------------------------
how can i make another skip: so my output would be:
1
2
3
4
5
11
12
13
14
18 ---> i skip from 15 to 17
19
20
tyvm.
| | Member | | Join Date: Mar 2007 Location: Manila,Philippines
Posts: 58
| | | re: need help doing a egrep and replace
oopsss. so sorry..
got the answers already..
tyvm..
|  | Similar Unix / Linux / BSD bytes | | | Forums
Visit our community forums for general discussions and latest on Bytes
/bytes/about
We are a network of experts and professionals in IT and software development that help one another with answers to tough questions and share insights.
Get the best answers to your questions from over 229,155 network members.
|