Connecting Tech Pros Worldwide Forums | Help | Site Map

need help doing a egrep and replace

Member
 
Join Date: Jan 2009
Posts: 68
#1: Mar 11 '09
Right now, I am using this line of code to get rid of data in a file called alarmNotification that I do not want:

Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\".*\"" $dbDir/alarmNotification.log
  2.  
but I am having some problem. An entry like the one below in the alarmNotification.log file is getting deleted when I only want a part of the entry to be deleted, keeping the rest of the line in tact

Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:21.01 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4600^A1236605421010^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY
  2.     at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  3.     at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
  4.     at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
  5.     at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
  6.     at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
  7.     at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
  8.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
  9.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
  10.     at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
  11.     at java.lang.Thread.run(Thread.java:619),^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  12.  
I want this entry to look like this after I correct the problem:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:21.01 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4600^A1236605421010^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 610712901:SUN:BUY^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  2.  

entries like these dont get effected at all by the egrep line:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:18.05 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4589^A1236605418050^A23605^A23603^A2^Am[0]=23601^A60^AProdcas01v2cas0102^APOATP/CASQuote^A" 0
  2. low debug 2009/3/9 8:30:19.52 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4590^A1236605419520^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:15:380 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(727). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=1171, delivered=570184, matched=0, discarded=742^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  3.  
Instead of an egrep, is there a global substitute or something that I could do.

thanks

gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#2: Mar 12 '09

re: need help doing a egrep and replace


I think u shuld change the regex for this.
As u use . also i thnk it is picking a wrong pattern.
Why cant u use like this
[a-z]a[z][a-z]\.[a-z]a-z]a-z]\. ==> for com.cboe



Raghu
Member
 
Join Date: Jan 2009
Posts: 68
#3: Mar 12 '09

re: need help doing a egrep and replace


So you mean have the regex like this:
Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\"[a-z]a[z][a-z]\.[a-z]a-z]a-z]\.*\"" $dbDir/alarmNotification.log
  2.  
is the syntax correct: the a[z] and the a-z]?

thanks
Member
 
Join Date: Jan 2009
Posts: 68
#4: Mar 12 '09

re: need help doing a egrep and replace


This is the problem actually:
This regex:
Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\".*\"" $dbDir/alarmNotification.log
  2.  
is looking for data between the two quotation marks:
Expand|Select|Wrap|Line Numbers
  1. ".*\"
  2.  
When I hate data like this:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4594^A1236605420470^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
  2.     at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  3.     at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
  4.     at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
  5.     at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
  6.     at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
  7.     at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
  8.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
  9.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
  10.     at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
  11.     at java.lang.Thread.run(Thread.java:619)
  12. ,^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  13.  
There are line break before each
Expand|Select|Wrap|Line Numbers
  1. at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  2.  
and so there is no closing quote after this line:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4594^A1236605420470^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
  2.  
I want my output to have everything, including the "at com.cboe..."
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4593^A1236605420470^A15904^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99) at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467) at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388) at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109) at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111) at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27) at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203) at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146) at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51) at java.lang.Thread.run(Thread.java:619),^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A
  2.  
but because they are different lines, the egrep doesnt select them. What can I use that will egrep between multiple lines from the beginning to the end of the quotation marks.

This data below works below it searches between the quotation marks and this is all on one line:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:19.53 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4592^A1236605419530^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:18:84 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(782). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=890, delivered=595531, matched=0, discarded=1542^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A0
  2.  
thanks
Member
 
Join Date: Jan 2009
Posts: 68
#5: Mar 13 '09

re: need help doing a egrep and replace


Hi,
Is someone able to help me with this problem

thanks
Member
 
Join Date: Jan 2009
Posts: 68
#6: Mar 13 '09

re: need help doing a egrep and replace


Is there something that I can use to search in a file over more then 1 line.
I believe that egrep only works line by line but can i search between lines?

thanks
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#7: Mar 16 '09

re: need help doing a egrep and replace


Hi,
Then i think you shuld wite your code using AWK.
I know AWK has this feature.

Raghu
Member
 
Join Date: Jan 2009
Posts: 68
#8: Mar 16 '09

re: need help doing a egrep and replace


Thanks gpraghuram,

Is there a code snippet you have that would help me

thanks
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#9: Mar 17 '09

re: need help doing a egrep and replace


Hi,
I dont have a example code with me.
But check this link
and look for this "Multiline Records" in the page.

Raghu
Member
 
Join Date: Jan 2009
Posts: 68
#10: Mar 17 '09

re: need help doing a egrep and replace


I checked out the link and I understand what they are saying:
Expand|Select|Wrap|Line Numbers
  1. Multiline Records
  2.  
  3. By default, the input record separator RS recognizes a newline as the marker between records. As is the norm in awk, this can be changed to allow for multiline records. When RS is set to the null string, the newline character always acts as a field separator, in addition to whatever value FS might have. 
  4.  
But how do I change RS so that it looks between two quotation marks:
Expand|Select|Wrap|Line Numbers
  1. ".*\"
  2.  
and the awk command that does that.

thanks
Member
 
Join Date: Jan 2009
Posts: 68
#11: Mar 17 '09

re: need help doing a egrep and replace


I used this snippet of code
Expand|Select|Wrap|Line Numbers
  1. awk -Flow print "low debug.*\".*\"" $dbDir/alarmNotification.log 
  2.  
I wanted the lines to be separated by the word low like in these entries:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 8:30:19.53 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4591^A1236605419530^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:16:775 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(18). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=0, delivered=582743, matched=0, discarded=760^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  2. low debug 2009/3/9 8:30:19.53 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4592^A1236605419530^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 08:30:18:84 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(782). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=890, delivered=595531, matched=0, discarded=1542^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  3. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4594^A1236605420470^A15906^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
  4.     at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  5.     at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
  6.     at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
  7.     at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
  8.     at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
  9.     at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
  10.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
  11.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
  12.     at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
  13.     at java.lang.Thread.run(Thread.java:619)
  14. ,^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  15. low debug 2009/3/9 8:30:20.47 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A4593^A1236605420470^A15904^A15902^A1^Am[0]=15901^Acom.cboe.exceptions.DataValidationException,IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY,com.cboe.exceptions.DataValidationException: IDL:exceptions/DataValidationException:1.0 com.cboe.exceptions.OrderBookTradableNotFoundException: No tradables in book at specified price: 639898851:SUN:BUY
  16.     at com.cboe.util.ExceptionBuilder.dataValidationException(ExceptionBuilder.java:99)
  17.     at com.cboe.businessServices.brokerService.BrokerProcessorBase.processUpdate(BrokerProcessorBase.java:2467)
  18.     at com.cboe.businessServices.brokerService.BrokerProcessorHybridImpl.processUpdate(BrokerProcessorHybridImpl.java:2388)
  19.     at com.cboe.businessServices.brokerService.AcceptQuoteUpdateCommand.doExecute(AcceptQuoteUpdateCommand.java:109)
  20.     at com.cboe.businessServices.brokerService.BrokerCommand.execute(BrokerCommand.java:111)
  21.     at com.cboe.server.commandProcessing.TradingClassBlockCommandSingleThreadImpl.acceptCommands(TradingClassBlockCommandSingleThreadImpl.java:27)
  22.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.processBlock(TradingClassBlockCommand.java:203)
  23.     at com.cboe.server.commandProcessing.TradingClassBlockCommand.execute(TradingClassBlockCommand.java:146)
  24.     at com.cboe.server.commandProcessing.TradingClassCommandQueueProcessor.run(TradingClassCommandQueueProcessor.java:51)
  25.     at java.lang.Thread.run(Thread.java:619)
  26. ,^AProdBC04x1HybridTradeServer1prdbc04a^AHybridTradeServer1^A" 0
  27.  
and I wanted it to print everything in the line:
Expand|Select|Wrap|Line Numbers
  1. low debug.*\".*\"
  2.  
from the file:
Expand|Select|Wrap|Line Numbers
  1. $dbDir/alarmNotification.log
  2.  
BUT i get these errors:
Expand|Select|Wrap|Line Numbers
  1. awk: syntax error near line 1
  2. awk: bailing out near line 1
  3. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-17/alarmNotificationTmp.txt (No such file or directory)
  4.  
What am i doing wrong?

thanks
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#12: Mar 18 '09

re: need help doing a egrep and replace


Instead of command line write a awk program which will be easy.
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. /*write code here for processing the line*/
  7. }
  8. END{
  9. }
Raghu
Member
 
Join Date: Jan 2009
Posts: 68
#13: Mar 18 '09

re: need help doing a egrep and replace


I created an AWK file called grepLogDebug.awk which has this code:
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. egrep "low debug.*\".*\"" $dbDir/alarmNotification.log;
  7. }
  8. END{
  9.  
and here is how I call the awk file:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk 
  2.  
and i get this error:
Expand|Select|Wrap|Line Numbers
  1. gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:6: egrep "low debug.*\".*\"" $dbDir/alarmNotification.log;
  2. gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:6:                                                   ^ parse error
  3. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-18/alarmNotificationTmp.txt (No such file or directory)
  4.  
Member
 
Join Date: Jan 2009
Posts: 68
#14: Mar 18 '09

re: need help doing a egrep and replace


I changed some of the code:

To call the AWK file:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk dbDir=$dbDir
  2.  
and in the AWK file:
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. printf "low debug.*\".*\"" > (dbDir "/alarmNotification.log");
  7. }
  8. END{
  9. }
  10.  
but when i run it,
it get stuck and doesnt finish creating the tables

thanks
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#15: Mar 19 '09

re: need help doing a egrep and replace


But where u are giving file name .
call like this
/infra/bin/gawk -f $workDir/grepLogDebug.awk dbDir=$dbDir < $IP_FILE

also you cant directly get the value of dbdir inside the AWK script and there is a special syntax to pass input arguments to the AWK.
Check in gawk manual

Raghu
Member
 
Join Date: Jan 2009
Posts: 68
#16: Mar 19 '09

re: need help doing a egrep and replace


is the $IP_FILE the name of the file that I am trying to print from:
so for me, the $IP_FILE should be: $dbDir/alarmNotification.log so should my call looks like this?
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log
  2.  
and then inside my grepLogDebug.awk file
is my print line correct because I am still getting errors:
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. printf "low debug.*\".*\"";
  7. }
  8. END{
  9.  
thanks
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#17: Mar 24 '09

re: need help doing a egrep and replace


printf is not ok.
printf in awk is similar to C syntax.

Expand|Select|Wrap|Line Numbers
  1. printf("low debug.*\".*\""); 
Raghu
Member
 
Join Date: Jan 2009
Posts: 68
#18: Mar 24 '09

re: need help doing a egrep and replace


I have changed the printf statement and now I get these errors when i try to create the tables:
Expand|Select|Wrap|Line Numbers
  1. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-24/alarmNotificationTmp.txt (No such file or directory)
  2.  
  3. fatal: error reading input file `stdin': I/O error
  4.  
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#19: Mar 25 '09

re: need help doing a egrep and replace


Hi,
I think this discissuion is going on for a long time.
Why cant u post the code and the way u run your script and i can check it.

Raghu
Member
 
Join Date: Jan 2009
Posts: 68
#20: Mar 25 '09

re: need help doing a egrep and replace


Hello Raghu,
Here is the code that I am running:
getAndArchiveIcsDbTest
Expand|Select|Wrap|Line Numbers
  1. #!/bin/ksh
  2. #/infra/sbt-support/supportTools/hsqldb/ics/setProperties
  3.  
  4. export javaDir=/usr/jdk/jdk1.5.0_06/bin
  5. export icsHost=deepfrzd
  6. export icsLogin=infra
  7. export workDir=/infra/sbt-support/supportTools/hsqldb/ics
  8. export tmpDir=/infra/sbt-support/supportTools/hsqldb/ics/tmp
  9. export AppRoot=/infra/sbt-support/supportTools/hsqldb/ics/DBConnection
  10. export hypDB=ICS_DB
  11. export archiveDir=/infra/sbt-support/supportTools/hsqldb/ics/tmp
  12. eval $(/infra/bin/EffectiveDate)
  13. export dbDir="$tmpDir/$WebDate"
  14.  
  15. remoteDbDir="/infra/sbt-support/supportTools/hsqldb/ics/archive/$1"
  16. remoteAlarmDir="/infra/sbt-support/supportTools/hsqldb/ics/archive/$1"
  17. previousDatesDir="/infra/sbt-support/supportTools/hsqldb/ics/archive"
  18.  
  19. mkdir $dbDir
  20.  
  21. $workDir/getIcsDbFiles.ksh $remoteDbDir $remoteAlarmDir
  22.  
  23. /infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
  24.  
  25. #egrep "low debug.*\".*\"" $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
  26.  
  27. $AppRoot/bin/runConvertMillis $dbDir
  28.  
  29. $AppRoot/bin/runDBConnectionHyp $AppRoot/etc/hypProperties $AppRoot/sql/createTextTables.sql $tmpDir/$WebDate/$hypDB
  30.  
  31. cd $tmpDir
  32. tar -cf $WebDate.tar $WebDate
  33. gzip -f $WebDate.tar
  34.  
  35. rm $WebDate.tar.gz
  36. mv $WebDate $1
  37. tar -cvf $1.tar $1
  38. gzip $1.tar
  39. mv $1.tar.gz $previousDatesDir
  40.  
  41. # mv -f $WebDate.tar.gz $archiveDir
  42. # rm -rf $WebDate
  43. # rmdir $WebDate
  44.  
grepLogDebug.awk
Expand|Select|Wrap|Line Numbers
  1. BEGIN{
  2. RS="";
  3. FS="low"
  4. }
  5. {
  6. printf("low debug.*\".*\"");
  7. }
  8. END{
  9. }
  10.  
here is a sample of the alarmNotification.log
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 10:33:42.03 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7089^A1236612822030^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 10:33:38:211 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(2819). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=1970, delivered=41296526, matched=0, discarded=2453375^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  2. low debug 2009/3/9 10:33:46.74 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7090^A1236612826740^A23906^A23904^A2^Am[0]=23902^A<FINE> <org.apache.activemq.broker.region.TopicSubscription> < Mon 2009/03/09 10:33:44:486 > <org.apache.activemq.broker.region.TopicSubscription.add> <Thread[ActiveMQ Transport: tcp:///127.0.0.1:64770,4,main]> Discarding state cleared, delta-discarded(16). TopicSubscription: consumer=ID:mdgc01a-47174-1236574999518-0:126:1:2, destination=topic:///ProdRecap/IDL:consumers/RecapConsumer:1.0---RecapLocalMD01---local, destinations=1, dispatchedQueue=1996, delivered=41322105, matched=0, discarded=2453391^AProdAMQBrokermdgc01a.out^AProdLogWatchermdgc01a^A" 0
  3. low debug 2009/3/9 10:33:46.86 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7091^A1236612826860^A19971^A19968^A3^Am[0]=19833^A160^AProdBC91x1HybridTradeServer1prdbc91b^AMarketDataQueue_11^Am[1]=19831^A160^AProdBC91x1HybridTradeServer1prdbc91b^AMarketDataQueue_11^A" 0
  4. low debug 2009/3/9 10:34:05.22 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A7094^A1236612845220^A15601^A13503^A1^Am[0]=13501^Acom.cboe.infrastructureServices.persistenceService.UniqueConstraintException,No exception message given,Delegating stack trace to the causing exception.
  5. java.sql.SQLException: ORA-00001: unique constraint (TRDSESS_10_OWNER.SBTORDERUK) violated
  6.  
  7.     at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
  8.     at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
  9.     at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
  10.     at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
  11.     at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
  12.     at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)
  13.     at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1167)
  14.     at oracle.jdbc.driver.OraclePreparedStatement.sendBatch(OraclePreparedStatement.java:3628)
  15.     at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:1119)
  16.     at com.objectwave.persist.RDBConnection.commit(RDBConnection.java:536)
  17.     at com.objectwave.persist.RDBBroker.commit(RDBBroker.java:132)
  18.     at com.objectwave.persist.BrokerChangeList.commitAll(BrokerChangeList.java:59)
  19.     at com.objectwave.persist.BrokerTransactionLog.commit(BrokerTransactionLog.java:88)
  20.     at com.objectwave.persist.BrokerTransactionLog.actualCommit(BrokerTransactionLog.java:53)
  21.     at com.objectwave.transactionalSupport.TransactionLog.commit(TransactionLog.java:298)
  22.     at com.objectwave.transactionalSupport.Session.commit(Session.java:51)
  23.     at com.cboe.infrastructureServices.foundationFramework.transactionManagement.BOSession.commit(BOSession.java:99)
  24.     at com.cboe.infrastructureServices.foundationFramework.utilities.Transaction.commit(Transaction.java:139)
  25.     at com.cboe.ohs.domain.OrdersQueueReader.processPersistOrderList(OrdersQueueReader.java:149)
  26.     at com.cboe.ohs.domain.OrdersQueueReader.addTransactionalBoundary(OrdersQueueReader.java:130)
  27.     at com.cboe.ohs.domain.OrdersQueueReader.dequeueAndaddTransactionBoundaries(OrdersQueueReader.java:63)
  28.     at com.cboe.ohs.domain.OrdersQueueReader.run(OrdersQueueReader.java:42)
  29.     at java.lang.Thread.run(Thread.java:619)
  30. ,^AProdBC10x1OHServerHybridprdbc10a^AOHServerHybrid^A" 0
  31.  
I call the getAndArchiveIcsDbTest on the command line with a date as an argument..for example:
getAndArchiveIcsDbTest 2009-03-09

That is when i get the errors that I posted earlier.

If I uncomment out the egrep statement in the getAndArchiveIcsDbTest file and comment out the /infra/bin/gawk statement, it runs with no errors, but it does not print out the low debug statements that have multiple lines, from the alarmNotification.log file, which is what I am trying to fix.
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#21: Mar 26 '09

re: need help doing a egrep and replace


I think problem is this line
Expand|Select|Wrap|Line Numbers
  1.  
  2. #/infra/bin/gawk -f $workDir/grepLogDebug.awk > $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir 
  3. #change the above line to this
  4.  
  5. /infra/bin/gawk -f $workDir/grepLogDebug.awk < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir 
  6.  
Raghu
Member
 
Join Date: Jan 2009
Posts: 68
#22: Mar 26 '09

re: need help doing a egrep and replace


The code change got rid of the fatal error:
Expand|Select|Wrap|Line Numbers
  1. fatal: error reading input file `stdin': I/O error
  2.  
but I still get this exception error:
Expand|Select|Wrap|Line Numbers
  1. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-26/alarmNotificationTmp.txt (No such file or directory)
  2.  
I do not get this error when I comment out the
/infra/bin/gawk statement and uncomment the egrep statement.

Thanks
Member
 
Join Date: Jan 2009
Posts: 68
#23: Mar 26 '09

re: need help doing a egrep and replace


Another error that I found is that this line:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk < $dbDir/alarmNotification.log
  2.  
creates the output of this, which is incorrect:
Expand|Select|Wrap|Line Numbers
  1. low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"low debug.*".*"
  2.  
The line which I was using before:
Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\".*\"" $dbDir/alarmNotification.log
  2.  
creates this output which is correct except for the issue I am trying to fix:
Expand|Select|Wrap|Line Numbers
  1. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A792^A1236577300600^A15806^A15402^A1^Am[0]=15401^A1^AProdfixcas08v2fix8^AALL^A" 0
  2. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A795^A1236577300600^A15404^A15402^A1^Am[0]=15401^A1^AProdfixcas08v2fix8^AALL^A" 0
  3. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A796^A1236577300600^A15806^A15402^A1^Am[0]=15401^A1^AProdfixcas15v2fix1b^AALL^A" 0
  4. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A797^A1236577300600^A15404^A15402^A1^Am[0]=15401^A1^AProdfixcas15v2fix1b^AALL^A" 0
  5. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A794^A1236577300600^A11329^A11327^A1^Am[0]=11326^A1^AProdfixcas08v2fix8^AALL^A" 0
  6. low debug 2009/3/9 0:41:40.61 ICSNotificationAlarm Prodics01ics0003 IC "1.0^AB^A793^A1236577300600^A10704^A10702^A1^Am[0]=10701^A1^AProdfixcas15v2fix1b^AALL^A" 0
  7.  
thanks
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#24: Mar 27 '09

re: need help doing a egrep and replace


In the awk you are only combinig the multiple lines and u are not processing anything?
Also u have the replace logic still in egrep?

Raghu
Member
 
Join Date: Jan 2009
Posts: 68
#25: Mar 27 '09

re: need help doing a egrep and replace


are you talking about processing the data in the grepLogDebug.awk file?

i tried the egrep statement in the grepLogDebug.awk file:
Expand|Select|Wrap|Line Numbers
  1. FS="low"
  2. }
  3. {
  4. printf("low debug.*\".*\"");
  5. egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotification.log;
  6. }
  7. END{
  8. }
  9.  
and I call the statement like this in the getAndArchiveIcsDbTest file:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk $WebDate < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
  2.  
but I get this error:
Expand|Select|Wrap|Line Numbers
  1. gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:7: egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotification.log;
  2. gawk: /infra/sbt-support/supportTools/hsqldb/ics/grepLogDebug.awk:7:                                                                                              ^ parse error
  3. unable to open file java.io.FileNotFoundException: /infra/sbt-support/supportTools/hsqldb/ics/tmp/2009-03-27/alarmNotificationTmp.txt (No such file or directory)
  4.  
I am not sure where I am supposed to process anything and what I should do with the replace logic that is in the egrep. I though that my grepLogDebug.awk file would make it so that I would be searching over multiple lines and then the egrep:
Expand|Select|Wrap|Line Numbers
  1. egrep "low debug.*\".*\"" /infra/sbt-support/supportTools/hsqldb/ics/tmp/$1/alarmNotification.log;
  2.  
would be able to get the info that I need and sed commands:
Expand|Select|Wrap|Line Numbers
  1. /infra/bin/gawk -f $workDir/grepLogDebug.awk $WebDate < $dbDir/alarmNotification.log | sed -e 's/low.* *"\(.*\)".*/\1/g' | egrep "1.0" | egrep -v "CORBA" | /infra/bin/gawk -f $workDir/generateTables.awk dbDir=$dbDir
  2.  
would do the replacing and work that I need done on the data that is captured from the grepLogDebug.awk file.

I am trying to figure out where I am going wrong because I have the logic correct for searching and replacing if all my data is within the same line. I didnt think that because the data is scattered over multiple lines, I would have this much trouble.
Member
 
Join Date: Mar 2007
Location: Manila,Philippines
Posts: 58
#26: Mar 28 '09

re: need help doing a egrep and replace


hi all.,


it's been a long time since i replied/post here. while having some trouble right now doing my script using solaris, i'm getting error:

"grep -hblcnsviw" pattern error


the usual grep is not working here.i hope you could me.


thanks.


coaxfiber
Member
 
Join Date: Mar 2007
Location: Manila,Philippines
Posts: 58
#27: Mar 28 '09

re: need help doing a egrep and replace


coaxfiber> grep -E -e -hblcnsviw 'MTP3b' Health_1_Mar26_10:33
grep: illegal option -- E
grep: illegal option -- e
Usage: grep -hblcnsviw pattern file . . .


coaxfiber> grep -E -e 'MTP3b' -e 'Rnsap' Health_1_Mar26_10:33
grep: illegal option -- E
grep: illegal option -- e
Usage: grep -hblcnsviw pattern file . . .
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#28: Mar 30 '09

re: need help doing a egrep and replace


HI,
Why cant u start a new thread instead of appending to an existing thread?

Raghu
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#29: Mar 30 '09

re: need help doing a egrep and replace


Hi,
If u want to call egrep from awk then u shuld use system command.
If u want to replace something inside awk then u shuld use sub or gsub.
Better get a awk manual to read abt it.

Raghu
Member
 
Join Date: Mar 2007
Location: Manila,Philippines
Posts: 58
#30: Apr 1 '09

re: need help doing a egrep and replace


hi, can you help me out why i'm not allowed to post new thread.. thanks..
gpraghuram's Avatar
Expert
 
Join Date: Mar 2007
Location: Chennai
Posts: 1,266
#31: Apr 2 '09

re: need help doing a egrep and replace


This shuld be answered by administrators.

Admins,
can u look into this issue.

Raghu
Member
 
Join Date: Mar 2007
Location: Manila,Philippines
Posts: 58
#32: Apr 2 '09

re: need help doing a egrep and replace


the contents of my scripts:

#!/usr/bin/ksh

integer x=1
while ((x<=20));

do

print $x;

if [ $x = 5 ]
then
x=10
fi
((x=x+1));

done
_____________
output:
1
2
3
4
5
11
12
13
14
15
16
17
18
19
20
-------------------------------

how can i make another skip: so my output would be:


1
2
3
4
5
11
12
13
14
18 ---> i skip from 15 to 17
19
20



tyvm.
Member
 
Join Date: Mar 2007
Location: Manila,Philippines
Posts: 58
#33: Apr 2 '09

re: need help doing a egrep and replace


oopsss. so sorry..

got the answers already..

tyvm..
Reply