473,396 Members | 1,804 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

home > topics > linux > questions > need help doing a egrep and replace

Join Bytes to post your question to a community of 473,396 software developers and data experts.

need help doing a egrep and replace

112 100+
Right now, I am using this line of code to get rid of data in a file called alarmNotification that I do not want:

Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\".*\"" $dbDir/alarmNotification.log
  2.  
but I am having some problem. An entry like the one below in the alarmNotification.log file is getting deleted when I only want a part of the entry to be deleted, keeping the rest of the line in tact

Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:21.01 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4600^A1236605421010^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY
  2.     at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  3.     at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
  4.     at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
  5.     at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
  6.     at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
  7.     at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
  8.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
  9.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
  10.     at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
  11.     at java.lang.Thread.run(Thread.java:619),^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  12.  
I want this entry to look like this after I correct the problem:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:21.01 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4600^A1236605421010^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  2.  

entries like these dont get effected at all by the egrep line:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:18.05 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4589^A1236605418050^A23605^A23603^A2^Am[0]=23601^A60^AProdcas01v2cas0102^APOATP/CASQuote^A" 0
  2. low debug 2009/3/9 8:30:19.52 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4590^A1236605419520^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:15:380 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(727). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=1171, delivered=570184, matched=0, discarded=742^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  3.  
Instead of an egrep, is there a global substitute or something that I could do.

thanks
Mar 11 '09 #1
32 3193
gpraghuram
1,275 Expert 1GB
I think u shuld change the regex for this.
As u use . also i thnk it is picking a wrong pattern.
Why cant u use like this
[a-z]a[z][a-z]\.[a-z]a-z]a-z]\. ==> for com.cboe



Raghu
Mar 12 '09 #2
ndedhia1
112 100+
So you mean have the regex like this:
Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\"[a-z]a[z][a-z]\.[a-z]a-z]a-z]\.*\"" $dbDir/alarmNotification.log
  2.  
is the syntax correct: the a[z] and the a-z]?

thanks
Mar 12 '09 #3
ndedhia1
112 100+
This is the problem actually:
This regex:
Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\".*\"" $dbDir/alarmNotification.log
  2.  
is looking for data between the two quotation marks:
Expand|Select|Wrap|Line Numbers
  1. ".*\"
  2.  
When I hate data like this:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4594^A1236605420470^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
  2.     at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  3.     at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
  4.     at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
  5.     at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
  6.     at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
  7.     at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
  8.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
  9.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
  10.     at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
  11.     at java.lang.Thread.run(Thread.java:619)
  12. ,^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  13.  
There are line break before each
Expand|Select|Wrap|Line Numbers
  1. at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  2.  
and so there is no closing quote after this line:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4594^A1236605420470^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
  2.  
I want my output to have everything, including the "at com.cboe..."
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4593^A1236605420470^A15904^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99) at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467) at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388) at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109) at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111) at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27) at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203) at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146) at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51) at java.lang.Thread.run(Thread.java:619),^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 
  2.  
but because they are different lines, the egrep doesnt select them. What can I use that will egrep between multiple lines from the beginning to the end of the quotation marks.

This data below works below it searches between the quotation marks and this is all on one line:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:19.53 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4592^A1236605419530^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:18:84 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(782). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=890, delivered=595531, matched=0, discarded=1542^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  2.  
thanks
Mar 12 '09 #4
ndedhia1
112 100+
Hi,
Is someone able to help me with this problem

thanks
Mar 13 '09 #5
ndedhia1
112 100+
Is there something that I can use to search in a file over more then 1 line.
I believe that egrep only works line by line but can i search between lines?

thanks
Mar 13 '09 #6
gpraghuram
1,275 Expert 1GB
Hi,
Then i think you shuld wite your code using AWK.
I know AWK has this feature.

Raghu
Mar 16 '09 #7
ndedhia1
112 100+
Thanks gpraghuram,

Is there a code snippet you have that would help me

thanks
Mar 16 '09 #8
gpraghuram
1,275 Expert 1GB
Hi,
I dont have a example code with me.
But check this link
and look for this "Multiline Records" in the page.

Raghu
Mar 17 '09 #9
ndedhia1
112 100+
I checked out the link and I understand what they are saying:
Expand|Select|Wrap|Line Numbers
  1. Multiline Records
  2.  
  3. By default, the input record separator RS recognizes a newline as the marker between records. As is the norm in awk, this can be changed to allow for multiline records. When RS is set to the null string, the newline character always acts as a field separator, in addition to whatever value FS might have. 
  4.  
But how do I change RS so that it looks between two quotation marks:
Expand|Select|Wrap|Line Numbers
  1. ".*\"
  2.  
and the awk command that does that.

thanks
Mar 17 '09 #10
ndedhia1
112 100+
I used this snippet of code
Expand|Select|Wrap|Line Numbers
  1. awk -Flow print "low debug.*\".*\"" $dbDir/alarmNotification.log 
  2.  
I wanted the lines to be separated by the word low like in these entries:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:19.53 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4591^A1236605419530^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:16:775 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(18). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=0, delivered=582743, matched=0, discarded=760^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  2. low debug 2009/3/9 8:30:19.53 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4592^A1236605419530^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:18:84 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(782). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=890, delivered=595531, matched=0, discarded=1542^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  3. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4594^A1236605420470^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
  4.     at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  5.     at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
  6.     at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
  7.     at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
  8.     at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
  9.     at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
  10.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
  11.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
  12.     at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
  13.     at java.lang.Thread.run(Thread.java:619)
  14. ,^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  15. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4593^A1236605420470^A15904^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
  16.     at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  17.     at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
  18.     at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
  19.     at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
  20.     at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
  21.     at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
  22.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
  23.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
  24.     at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
  25.     at java.lang.Thread.run(Thread.java:619)
  26. ,^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  27.  
and I wanted it to print everything in the line:
Expand|Select|Wrap|Line Numbers
  1. low debug.*\".*\"
  2.  
from the file:
Expand|Select|Wrap|Line Numbers
  1. $dbDir/alarmNotification.log
  2.  
BUT i get these errors:
Expand|Select|Wrap|Line Numbers
  1. awk: syntax error near line 1
  2. awk: bailing out near line 1
  3. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-17/alarmNotificationTmp.txt (No such file or directory)
  4.  
What am i doing wrong?

thanks
Mar 17 '09 #11
gpraghuram
1,275 Expert 1GB
Instead of command line write a awk program which will be easy.
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. /*write code here for processing the line*/
  7. }
  8. END{
  9. }
Raghu
Mar 18 '09 #12
ndedhia1
112 100+
I created an AWK file called grepLogDebug.awk which has this code:
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. egrep "low debug.*\".*\"" $dbDir/alarmNotification.log;
  7. }
  8. END{
  9.  
and here is how I call the awk file:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk 
  2.  
and i get this error:
Expand|Select|Wrap|Line Numbers
  1. gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:6: egrep "low debug.*\".*\"" $dbDir/alarmNotification.log;
  2. gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:6:                                                   ^ parse error
  3. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-18/alarmNotificationTmp.txt (No such file or directory)
  4.  
Mar 18 '09 #13
ndedhia1
112 100+
I changed some of the code:

To call the AWK file:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk dbDir=$dbDir
  2.  
and in the AWK file:
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. printf "low debug.*\".*\"" > (dbDir "/alarmNotification.log");
  7. }
  8. END{
  9. }
  10.  
but when i run it,
it get stuck and doesnt finish creating the tables

thanks
Mar 18 '09 #14
gpraghuram
1,275 Expert 1GB
But where u are giving file name .
call like this
/infra/bin/gawk -f $workDir/grepLogDebug.awk dbDir=$dbDir < $IP_FILE

also you cant directly get the value of dbdir inside the AWK script and there is a special syntax to pass input arguments to the AWK.
Check in gawk manual

Raghu
Mar 19 '09 #15
ndedhia1
112 100+
is the $IP_FILE the name of the file that I am trying to print from:
so for me, the $IP_FILE should be: $dbDir/alarmNotification.log so should my call looks like this?
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log
  2.  
and then inside my grepLogDebug.awk file
is my print line correct because I am still getting errors:
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. printf "low debug.*\".*\"";
  7. }
  8. END{
  9.  
thanks
Mar 19 '09 #16
gpraghuram
1,275 Expert 1GB
printf is not ok.
printf in awk is similar to C syntax.

Expand|Select|Wrap|Line Numbers
  1. printf("low debug.*\".*\""); 
Raghu
Mar 24 '09 #17
ndedhia1
112 100+
I have changed the printf statement and now I get these errors when i try to create the tables:
Expand|Select|Wrap|Line Numbers
  1. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-24/alarmNotificationTmp.txt (No such file or directory)
  2.  
  3. fatal: error reading input file `stdin': I/O error
  4.  
Mar 24 '09 #18
gpraghuram
1,275 Expert 1GB
Hi,
I think this discissuion is going on for a long time.
Why cant u post the code and the way u run your script and i can check it.

Raghu
Mar 25 '09 #19
ndedhia1
112 100+
Hello Raghu,
Here is the code that I am running:
getAndArchiveIcsDbTest
Expand|Select|Wrap|Line Numbers
  1. #!/bin/ksh
  2. #/infra/sbt-support/supportTools/hsqldb/ics/setProperties
  3.  
  4. export javaDir=/usr/jdk/jdk1.5.0_06/bin
  5. export icsHost=deepfrzd
  6. export icsLogin=infra
  7. export workDir=/infra/sbt-support/supportTools/hsqldb/ics
  8. export tmpDir=/infra/sbt-support/supportTools/hsqldb/ics/tmp
  9. export AppRoot=/infra/sbt-support/supportTools/hsqldb/ics/DBConnection
  10. export hypDB=ICS_DB
  11. export archiveDir=/infra/sbt-support/supportTools/hsqldb/ics/tmp
  12. eval $(/infra/bin/EffectiveDate)
  13. export dbDir="$tmpDir/$WebDate"
  14.  
  15. remoteDbDir="/infra/sbt-support/supportTools/hsqldb/ics/archive/$1"
  16. remoteAlarmDir="/infra/sbt-support/supportTools/hsqldb/ics/archive/$1"
  17. previousDatesDir="/infra/sbt-support/supportTools/hsqldb/ics/archive"
  18.  
  19. mkdir $dbDir
  20.  
  21. $workDir/getIcsDbFiles.ksh $remoteDbDir $remoteAlarmDir
  22.  
  23. /infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
  24.  
  25. #egrep "low debug.*\".*\"" $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
  26.  
  27. $AppRoot/bin/runConvertMillis $dbDir
  28.  
  29. $AppRoot/bin/runDBConnectionHyp $AppRoot/etc/hypProperties $AppRoot/sql/createTextTables.sql $tmpDir/$WebDate/$hypDB
  30.  
  31. cd $tmpDir
  32. tar -cf $WebDate.tar $WebDate
  33. gzip -f $WebDate.tar
  34.  
  35. rm $WebDate.tar.gz
  36. mv $WebDate $1
  37. tar -cvf $1.tar $1
  38. gzip $1.tar
  39. mv $1.tar.gz $previousDatesDir
  40.  
  41. # mv -f $WebDate.tar.gz $archiveDir
  42. # rm -rf $WebDate
  43. # rmdir $WebDate
  44.  
grepLogDebug.awk
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. printf("low debug.*\".*\"");
  7. }
  8. END{
  9. }
  10.  
here is a sample of the alarmNotification.log
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 10:33:42.03 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7089^A1236612822030^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 10:33:38:211 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(2819). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=1970, delivered=41296526, matched=0, discarded=2453375^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  2. low debug 2009/3/9 10:33:46.74 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7090^A1236612826740^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 10:33:44:486 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(16). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=1996, delivered=41322105, matched=0, discarded=2453391^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  3. low debug 2009/3/9 10:33:46.86 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7091^A1236612826860^A19971^A19968^A3^Am[0]=19833^A160^AProdBC91x1HybridTradeServer1prdbc91b^AMarketDataQueue_11^Am[1]=19831^A160^AProdBC91x1HybridTradeServer1prdbc91b^AMarketDataQueue_11^A" 0
  4. low debug 2009/3/9 10:34:05.22 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7094^A1236612845220^A15601^A13503^A1^Am[0]=13501^Acom.cboe.infrastructureServices.persistenceService.UniqueConstraintException,No exception message given,Delegating stack trace to the causing exception.
  5. java.sql.SQLException: ORA-00001: unique constraint (TRDSESS_10_OWNER.SBTORDERUK) violated
  6.  
  7.     at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
  8.     at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
  9.     at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
  10.     at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
  11.     at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
  12.     at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)
  13.     at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1167)
  14.     at oracle.jdbc.driver.OraclePreparedStatement.sendBatch(OraclePreparedStatement.java:3628)
  15.     at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:1119)
  16.     at com.objectwave.persist.RDBConnection.commit(RDBConnection.java:536)
  17.     at com.objectwave.persist.RDBBroker.commit(RDBBroker.java:132)
  18.     at com.objectwave.persist.BrokerChangeList.commitAll(BrokerChangeList.java:59)
  19.     at com.objectwave.persist.BrokerTransactionLog.commit(BrokerTransactionLog.java:88)
  20.     at com.objectwave.persist.BrokerTransactionLog.actualCommit(BrokerTransactionLog.java:53)
  21.     at com.objectwave.transactionalSupport.TransactionLog.commit(TransactionLog.java:298)
  22.     at com.objectwave.transactionalSupport.Session.commit(Session.java:51)
  23.     at com.cboe.infrastructureServices.foundationFramework.transactionManagement.BOSession.commit(BOSession.java:99)
  24.     at com.cboe.infrastructureServices.foundationFramework.utilities.Transaction.commit(Transaction.java:139)
  25.     at com.cboe.ohs.domain.OrdersQueueReader.processPersistOrderList(OrdersQueueReader.java:149)
  26.     at com.cboe.ohs.domain.OrdersQueueReader.addTransactionalBoundary(OrdersQueueReader.java:130)
  27.     at com.cboe.ohs.domain.OrdersQueueReader.dequeueAndaddTransactionBoundaries(OrdersQueueReader.java:63)
  28.     at com.cboe.ohs.domain.OrdersQueueReader.run(OrdersQueueReader.java:42)
  29.     at java.lang.Thread.run(Thread.java:619)
  30. ,^AProdBC10x1OHServerHybridprdbc10a^AOHServerHybrid^A" 0
  31.  
I call the getAndArchiveIcsDbTest on the command line with a date as an argument..for example:
getAndArchiveIcsDbTest 2009-03-09

That is when i get the errors that I posted earlier.

If I uncomment out the egrep statement in the getAndArchiveIcsDbTest file and comment out the /infra/bin/gawk statement, it runs with no errors, but it does not print out the low debug statements that have multiple lines, from the alarmNotification.log file, which is what I am trying to fix.
Mar 25 '09 #20
gpraghuram
1,275 Expert 1GB
I think problem is this line
Expand|Select|Wrap|Line Numbers
  1.  
  2. #/infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir 
  3. #change the above line to this
  4.  
  5. /infra/bin/gawk -f $workDir/grepLogDebug.awk < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir 
  6.  
Raghu
Mar 26 '09 #21
ndedhia1
112 100+
The code change got rid of the fatal error:
Expand|Select|Wrap|Line Numbers
  1. fatal: error reading input file `stdin': I/O error
  2.  
but I still get this exception error:
Expand|Select|Wrap|Line Numbers
  1. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-26/alarmNotificationTmp.txt (No such file or directory)
  2.  
I do not get this error when I comment out the
/infra/bin/gawk statement and uncomment the egrep statement.

Thanks
Mar 26 '09 #22
ndedhia1
112 100+
Another error that I found is that this line:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk < $dbDir/alarmNotification.log
  2.  
creates the output of this, which is incorrect:
Expand|Select|Wrap|Line Numbers
  1. low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"
  2.  
The line which I was using before:
Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\".*\"" $dbDir/alarmNotification.log
  2.  
creates this output which is correct except for the issue I am trying to fix:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A792^A1236577300600^A15806^A15402^A1^Am[0]=15401^A1^AProdfixcas08v2fix8^AALL^A" 0
  2. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A795^A1236577300600^A15404^A15402^A1^Am[0]=15401^A1^AProdfixcas08v2fix8^AALL^A" 0
  3. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A796^A1236577300600^A15806^A15402^A1^Am[0]=15401^A1^AProdfixcas15v2fix1b^AALL^A" 0
  4. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A797^A1236577300600^A15404^A15402^A1^Am[0]=15401^A1^AProdfixcas15v2fix1b^AALL^A" 0
  5. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A794^A1236577300600^A11329^A11327^A1^Am[0]=11326^A1^AProdfixcas08v2fix8^AALL^A" 0
  6. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A793^A1236577300600^A10704^A10702^A1^Am[0]=10701^A1^AProdfixcas15v2fix1b^AALL^A" 0
  7.  
thanks
Mar 26 '09 #23
gpraghuram
1,275 Expert 1GB
In the awk you are only combinig the multiple lines and u are not processing anything?
Also u have the replace logic still in egrep?

Raghu
Mar 27 '09 #24
ndedhia1
112 100+
are you talking about processing the data in the grepLogDebug.awk file?

i tried the egrep statement in the grepLogDebug.awk file:
Expand|Select|Wrap|Line Numbers
  1. FS="low"
  2. }
  3. {
  4. printf("low debug.*\".*\"");
  5. egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotification.log;
  6. }
  7. END{
  8. }
  9.  
and I call the statement like this in the getAndArchiveIcsDbTest file:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk $WebDate < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
  2.  
but I get this error:
Expand|Select|Wrap|Line Numbers
  1. gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:7: egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotification.log;
  2. gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:7:                                                                                              ^ parse error
  3. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-27/alarmNotificationTmp.txt (No such file or directory)
  4.  
I am not sure where I am supposed to process anything and what I should do with the replace logic that is in the egrep. I though that my grepLogDebug.awk file would make it so that I would be searching over multiple lines and then the egrep:
Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotification.log;
  2.  
would be able to get the info that I need and sed commands:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk $WebDate < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
  2.  
would do the replacing and work that I need done on the data that is captured from the grepLogDebug.awk file.

I am trying to figure out where I am going wrong because I have the logic correct for searching and replacing if all my data is within the same line. I didnt think that because the data is scattered over multiple lines, I would have this much trouble.
Mar 27 '09 #25
hi all.,


it's been a long time since i replied/post here. while having some trouble right now doing my script using solaris, i'm getting error:

"grep -hblcnsviw" pattern error


the usual grep is not working here.i hope you could me.


thanks.


coaxfiber
Mar 28 '09 #26
coaxfiber> grep -E -e -hblcnsviw 'MTP3b' Health_1_Mar26_10:33
grep: illegal option -- E
grep: illegal option -- e
Usage: grep -hblcnsviw pattern file . . .


coaxfiber> grep -E -e 'MTP3b' -e 'Rnsap' Health_1_Mar26_10:33
grep: illegal option -- E
grep: illegal option -- e
Usage: grep -hblcnsviw pattern file . . .
Mar 28 '09 #27
gpraghuram
1,275 Expert 1GB
HI,
Why cant u start a new thread instead of appending to an existing thread?

Raghu
Mar 29 '09 #28
gpraghuram
1,275 Expert 1GB
Hi,
If u want to call egrep from awk then u shuld use system command.
If u want to replace something inside awk then u shuld use sub or gsub.
Better get a awk manual to read abt it.

Raghu
Mar 29 '09 #29
hi, can you help me out why i'm not allowed to post new thread.. thanks..
Apr 1 '09 #30
gpraghuram
1,275 Expert 1GB
This shuld be answered by administrators.

Admins,
can u look into this issue.

Raghu
Apr 2 '09 #31
the contents of my scripts:

#!/usr/bin/ksh

integer x=1
while ((x<=20));

do

print $x;

if [ $x = 5 ]
then
x=10
fi
((x=x+1));

done
_____________
output:
1
2
3
4
5
11
12
13
14
15
16
17
18
19
20
-------------------------------

how can i make another skip: so my output would be:


1
2
3
4
5
11
12
13
14
18 ---> i skip from 15 to 17
19
20



tyvm.
Apr 2 '09 #32
oopsss. so sorry..

got the answers already..

tyvm..
Apr 2 '09 #33

Sign in to post your reply or Sign up for a free account.

Similar topics

9
Need some help with design of tables/views
by: sk | last post by:
I have an applicaton in which I collect data for different parameters for a set of devices. The data are entered into a single table, each set of name, value pairs time-stamped and associated with...
Microsoft SQL Server
2
Need help change a particular word in an XML doc
by: Larry C | last post by:
Hello, I have an XML doc and I am trying to write a utility thath will allow me to change a particular word with my "title" section of theXML doc. I would like to look for a "word" in the title...
.NET Framework
19
Need better argument for using bound forms
by: James Fortune | last post by:
I have a lot of respect for David Fenton and Allen Browne, but I don't understand why people who know how to write code to completely replace a front end do not write something that will automate...
Microsoft Access / VBA
3
First argument lost when doing exec
by: Goh, Yong Kwang | last post by:
I'm currently using execvp() for my lab program and I realise that the first argument (args) I pass to execvp is always not passed to the file ("/bin/ls") being executed. I'm using gcc compiler...
C / C++
3
Need pixie dust for building Python 2.4 curses module on Solaris 8
by: skip | last post by:
I'm having no success building the curses module on Solaris 8 (yes, I know it's ancient - advancing the state-of-the-art is not yet an option) for Python 2.4. Sun provides an apparently ancient...
Python
3
Need some help with replace()
by: Roy W. Andersen | last post by:
Hi, I need to do some replace-calls on certain strings in order to replace smiley glyphs and other keywords with graphical icons on the client. Unfortunately, my knowledge of regular expressions...
Javascript
1
Problem with searching files with egrep
by: SSJVEGETA | last post by:
Hello, everybody. I have read some examples and manuals for the egrep command for Linux and I don't know if this egrep command is right for the particular files I am searching for. Here is what the...
Linux
19
Need of a code snipet which converts mm/dd/yy to dd/mm/yy
by: santanu mishra | last post by:
Hi , I am stuck with a requirement from my client to change the date format from mm/dd/yy to dd/mm/yy .If any body can help me out with this regard as its very much urgent. Regards,...
Javascript
1
I need some hints please
by: Luron31 | last post by:
Hello everyone, im doing a project and im really stuck on it... i dont want answers. i have a very guilty conscience. i just need some hints as to what im doing wrong or what i could do...
Java
0
Merging data from multiple Excel files
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
Data Management
0
Migrating Website to Cloud - Emmanuel Katto
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
General
1
Is that possible of reading the .csv file in column wise and the column have different lengths ?
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
C / C++
0
How to build RAID in BIOS?
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
Computer Hardware
0
marktang
What is ONU?
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
General
0
Oralloy
Problem With Comparison Operator <=> in G++
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
C / C++
0
jinu1996
Maximizing Business Potential: The Nexus of Website Design and Digital Marketing
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
Online Marketing
0
The easy way to turn off automatic updates for Windows 10/11
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
Windows Server
0
agi2029
AI Job Threat for Devs
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
Career Advice

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.

BYTES.COM © 2024
About Bytes
Terms Of Use
Privacy Policy
Sitemap
Advertise on Bytes:
Post a Job
Sponsored Posts
Platinum & Gold Sponsors
Hire Now!