Perl : HEX parameter handling

[somyasagar]

Hi,

Please tell how to write :A script which reads a file and search for some words like ABC or FGH in the file text and put some variable after their occurance.That variable is configurable HEX parameter which can be configured through config file or can be given as a command line parameter.

Thanks

[miller]

<RANT>
No Thanks

If you really would like some help you should appear to have put at least some effort into the problem before posting on this forum. Show us some code. Show us that you have at least thought about possible solutions and have simply run into a problem. All you've done thus far is demonstrated that you can copy and paste a homework assignment.
</RANT>

[somyasagar]

Hi All ,

I am able to write some code for the above said problem.
The code is given below .
But i still have 2 problems.

1. There is occurance of some blank lines when i execute the script, I need to remove those blank lines. and

2. I need to compare the two lines of a given file ,say

BHS|^~\&|NVA |
MSH|^~\&|NVA |
MSH|^~\&|NVB |
EVN|P03|20060702 |
PID| |5360738 |

If there is occurance of MSH segment after BHS segment then change the first occurance of MSH to MSD.


Code:

#!/usr/local/bin/perl -w


# This Perl program in classic programming style changes
# the string "headerOld" to "headerNew" in all files
# specified on the command line.

@headerOld=('BHS','BTS','FTS');
@headerNew=("\nABHS","\nBBTS","\nFFTS");


for ($i=0;$i<3 ;$i++) {

$nchanges = 0;
undef $/;

foreach $file (@ARGV) {
if (! open(INPUT,"<$file") ) {
print STDERR "Can't open input file $bakfile\n";
next;
}

# Read input file as one long record.
$data=<INPUT>;
close INPUT;

# Replaces the headerOld with headerNew in the $data content.
if ($data =~ s/$headerOld[$i]/$headerNew[$i]/gi) {
$bakfile = "$file.bak";
# Abort if can't backup original or output.
if (! rename($file,$bakfile)) {
die "Can't rename $file $!";
}
if (! open(OUTPUT,">$file") ) {
die "Can't open output file $file\n";
}
$data =~ s/\n+/\n/g;

print OUTPUT $data;
close OUTPUT;
$nchanges++;

}
else { print STDERR "$file not changed\n"; }
}
}

print STDERR "$nchanges files changed.\n";

exit(0);

[miller]

Some initial comments and changes to your code:

1) Format your code. It helps us read it faster, and even more importantly, it will help you as well.

2) You should perform all transformations on each file, instead of each transformation on all files. In other words, limit hte number of times that you are opening and closing a file. Currently, you open and close each file for every transformation. Along with slowing down your performance by an order of magnitude, this also adds risk with regard to race conditions. Avoid this.

3) "use strict;" Always include this line in your file. It will help you out more than you could ever dream, although I see that you are using -w as well, which is good. I found one variable misspelling immediately upon making this change.

4) "undef $/;" is bad. use "local $/;" instead. Read about why at http://perldoc.perl.org/perlvar.html (Search for $/)

5) Instead of "print STDERR", try using "warn" instead. See http://perldoc.perl.org/functions/warn.html

6) What's with the return characters in @headerNew? Do you really want those to be there? I'm guessing not.

7) $i<3 is a bad idea. Instead use the number of elements in @headerOld as your limitting factor.

8) exit(0) is nice to do, but it's kind of redundant. Just do 1;\n\n__END__

#!/usr/local/bin/perl -w
 
# This Perl program in classic programming style changes
# the string "headerOld" to "headerNew" in all files
# specified on the command line.
 
use strict;
 
my @headerOld = ('BHS','BTS','FTS');
my @headerNew = ('ABHS','BBTS','FFTS');
 
my $nchanges = 0;
 
foreach my $file (@ARGV) {
    open(INPUT, "< $file") or warn("Can't open input file $file\n"), next;
 
    # Read input file as one long record.
    local $/;
 
    my $data = <INPUT>;
    close INPUT;
 
    my $hasChanged = 0;
 
    for (my $i=0; $i<@headerOld; $i++) {
        # Replaces the headerOld with headerNew in the $data content.
        $hasChanged += $data =~ s/$headerOld[$i]/$headerNew[$i]/gi;
    }
 
    if ($hasChanged) {
        # Wippee!
        my $bakfile = "$file.bak";
 
        # Abort if can't backup original or output.
        rename($file, $bakfile) or die "Can't rename $file $!";
 
        open(OUTPUT, "> $file") or die "Can't open output file $file\n";
 
        $data =~ s/\n+/\n/g;
 
        print OUTPUT $data;
        close OUTPUT;
 
        $nchanges++;
 
    } else {
        warn "$file not changed\n";
    }
}
 
warn "$nchanges files changed.\n";
 
1;
 
__END__
 

Ok, now that we've gone through all that cleanup, we come to your real problem. You currently describe two different things that you're trying to accomplish:

Goal 1) There is occurance of some blank lines when i execute the script, I need to remove those blank lines. and

Are these blank lines "added" by your script? or are they originally in the text file and you want to parse them out? I believe that if the former is true, then I've fixed your problem by editing @headerNew.

Goal 2) Duplicate MSH

I need to compare the two lines of a given file ,say

BHS|^~\&|NVA |
MSH|^~\&|NVA |
MSH|^~\&|NVB |
EVN|P03|20060702 |
PID| |5360738 |

If there is occurance of MSH segment after BHS segment then change the first occurance of MSH to MSD.

I believe you meant:

If there are TWO MSH segments after a BHS segment then change the first occurance of MSH to MSD.

Is this true?


Goal 3) Direct translation of some text

Change the following values:
'BHS' => 'ABHS'
'BTS' => 'BBTS'
'FTS' => 'FFTS'

Do you want to change all of these values regardless of when they occur? Do these values only occur at the beginning of a record (or line)?

Clarify these questions and I'll help you further.

[somyasagar]

First of all I would like to thank you ,Miller for your help…
It has boosted up my interest to put more efforts in learning and working with PERL.

Let me explain your findings one by one:

Goal 1) There is occurance of some blank lines when i execute the script, I need to remove those blank lines. and

Note: Yes these lines are added when I run the script.

Goal 2) Duplicate MSH

I believe you meant:

If there are TWO MSH segments after a BHS segment then change the first occurance of MSH to MSD.

Is this true?

Note: In the message there could be more then two MHS as well. But the first occurrence after BHS need to be changed .say MSH to /rMSH.
Sample message:

BHS|^~\&|NVA |
MSH|^~\&|NVA |
MSH|^~\&|NVB |
EVN|P03|20060702 |
PID| |5360738 |
BHS|^~\&|NVA |
MSH|^~\&|NVA |
MSH|^~\&|NVB |
MSH|^~\&|NVA |
MSH|^~\&|NVB |
EVN|P03|20060702 |
PID| |5360738 |

Goal 3) Direct translation of some text

Change the following values:
'BHS' => 'ABHS'
'BTS' => 'BBTS'
'FTS' => 'FFTS'

Do you want to change all of these values regardless of when they occur? Do these values only occur at the beginning of a record (or line)?


Note: Yes these values are changed regardless of when they occur.
There could be thousands of MSH and Hundreds of BHS.

The message is read as different records and that is achieved by doing the above replacements , they act as record separators.
Say …If I replace BHS by \nBHS the there is addition of “0d 0a ” in hex.
And "0d 0a" becomes the record separotor.
The same way I need to change for FTS , BTS and the MSH(first occurrence after BHS).
The separator is need to be kept same for all the replacements.

[somyasagar]

Please help me out with the MSH replacement function....
I am not able to figure it out.

Sorry but my code is not working at all...

[miller]

Hi All ,
2. I need to compare the two lines of a given file ,say

BHS|^~\&|NVA |
MSH|^~\&|NVA |
MSH|^~\&|NVB |
EVN|P03|20060702 |
PID| |5360738 |

If there is occurance of MSH segment after BHS segment then change the first occurance of MSH to MSD.

I'm editing code for you right now. Should only take me a 10 minutes. I'm still unsure why you said first in the above statement though. In the end I'm going to end up making up my own rules concerning what you might have intended, and you can simply use the code as starting point since I really don't understand your specification.

[miller]

use File::Basename qw(dirname);
use File::Temp;
use POSIX qw(strftime);
 
use strict;
 
# Goal 1
# - If there is an MSH segment immediately after a BHS segment, then replace it with MSD
# - Note: Segments are defined as the letters beginning a line
 
# Goal 2
# - Rename Segments to a new type.
my %translate = (
    'BHS' => 'ABHS',
    'BTS' => 'BBTS',
    'FTS' => 'FFTS',
);
my $translateKeys = join('|', keys %translate);
 
my $nchanges = 0;
 
foreach my $file (@ARGV) {
    open(INPUT, "< $file") or warn("Can't open input file $file\n"), next;
 
    print "processing: $file\n";
 
    # This assures that the new file will actually be unique.
    my $tmp = new File::Temp(
        DIR        => dirname($file),
        UNLINK    => 0,
        SUFFIX    => '.tmp',
    );
 
    print "\tSaving new data to: " . $tmp->filename . "\n";
 
    my $changes = 0;
 
    # Process the file one line at a time
    while (my $queue = <INPUT>) {
        # Might end up with queue of lines to process
        my @lines = ($queue);
 
        # Begins with BHS (Goal 1 Code)
        # - Note: Cannot be at the end of the file.
        if ($queue =~ m/^BHS/ && ! eof(INPUT)) {
            $queue = <INPUT>;
            $changes += $queue =~ s/^MSH/MSD/;
            push @lines, $queue;
        }
 
        # Process Possibly Multiple lines
        foreach my $line (@lines) {
            # Translate segments (Goal 2 Code)
            $changes += $line =~ s/^($translateKeys)/$translate{$1}/g;
 
            # Output Results.
            print $tmp $line;
        }
    }
 
    close(INPUT) or die "close $file: $!";
    $tmp->close() or die "close $tmp: $!";
 
    print "\tChanges Found: $changes\n";
    $nchanges++ if $changes;
 
    # Backup Old File
    # - Alternatively, you can choose to just delete the old file.
    # unlink($file) or die "unlink $file: $!";
    my $bakfile = $file . strftime(".%Y%m%d.%H%M%S", localtime()) . '.bak';
    rename($file, $bakfile) or die "can't rename $file: $!";
 
    # Move Processed Results to old file location
    rename($tmp->filename, $file) or die "can't rename $tmp to $file: $!";
}
 
warn "$nchanges files changed.\n";
 
1;
 
__END__
 

The above code does many of the things that you needed. Even if this doesn't fit your specifications exactly, it at least should give you an example of a better way of processing files line by line and as a group of lines. I've also taken into account many things that your code was failing to do. I leave it as an exercise for yourself to determine what those changes were and why.

Additionally, I also added dependencies to three CPAN modules. I believe that those are all part of the default installation of perl, but if not, then they are easily added from http://cpan.perl.org. Alternatively, they could be removed from the script entirely, but you would have to discover for yourself how to do this.

[somyasagar]

Thanks Miller.

I used first becoz ur the first one who has replied to my first post on any forum.
This is my first experience of being associated with any group online.
Morover very first experience working with PERL.

Thanks again ...

I m going to finish the exercise given by u. If any problem persists i will again get back to u.

regards...

[somyasagar]

Hi Miller,

This is in reference to my post #3, the only thing i wasn't able to add is the liberty of replacing MSH(first occurance after BHS in it) otherwise things are working fine .
The changes u made in post #4 are the best suitable for me.

In code we can the scenerio as :

my @headerOld = ('BHS','BTS','FTS');
my @headerNew = ("\nBHS","\nBTS","\nFTS");

In the same way i want to replace MSH(first occurance after BHS ) with "\nMSH".

The replacement prefix value can be anything like "\n\t" etc.

Please help me with the current funtionality addition.The test data can be same as in post post #5.

It is my mistake that i am not able to write the best explainatory specification for the problem.

regards...

[somyasagar]

Hi
The functionality of changing MSH is required at every step not the only the 1st occurance after BHS...so done. Thanks

Some more complexities i am finding in my script.

1) Can i parse any HEX character in perl and check its value and replace by a different HEX in any record.

e.g This is Perl scripts developer community.

In any HEX editor if i open this message then i found 20 against space .
Question :How can i search for a HEX character in any given file.

2) The HEX replacement should be done between given segments only (say between FHS[end of record ] separated by HEX 0a and BHS OR BHS [end of record]separated by HEX 0a and MSH ).And i kno the separator between is HEX 0a.I need to change from 0a to HEX 0d 0d.
How to replace frm 0a to 0d 0d in HEX.
If there is no separator(Here HEX 0a) in between the given segments then script should promt error.:Wrong message!!!!

3)Can i make this HEX parameter ( say 0d 0d) configurable ,to store in a variable and change in the script.

4)The script should exit at that point with proper logging.

I am totally stuck with this requirment of the script.
Please help.

[miller]

Hello somyasager.

I'm grateful that you took my initial advice to attempt a solution before posting a problem to heart. It appears that you've put a lot of effort into the goal of this script, which is great.

Unfortunately, I (personally) cannot help you anymore with this. If it were just a bug that you needed help decoding, of course. If it were clear specifications that you needed help coming up with an attack method, then of course. But at this point it sounds like like I'm investing time into educating you concerning a problem that I cannot get a clear understanding of compounded by a language barrier.

My second script that I posted here should give you an example of how to write a perl code that will read in a file, translate text on a per line or multi line basis, and output those results back to their original file.

Given that information, all you need to learn how to do is use regular expressions or some other logic to make the transformations that you require. Unfortunately I cannot read your mind to help with you them at this point.

Anyway, I'm only bothering to respond since I've invested a couple of hours to helping you already, and I did not want you to be waiting futilly for another response.

...

I've attempted to quickly answer your lateast questions, but I'm still confused by what you want. To answer your basic question, yes, I'm certain that whatever you're attempting to do is possible. Basically you need to use regular expressions to detect when you want to attempt a search and replace, and regular expressions to do the actually search and replace. Also, I'm pretty sure that you mean hex sequence and not hex characters, but once again, I don't know.

Anyway, good luck in your quest.

[somyasagar]

Hi ,

I am writing a script that changes the Hex Sequence in a given file:
The code is as follows:
################################################## ###########

#!/usr/bin/perl -w

#Perl script to change the HEX sequence from a given value to new value.
#Here "0a" in replaced by "0a 0b"

package FooterHeader;

require Exporter;
our @ISA = qw(Exporter);
our @EXPORT = qw(
sub FHSBHS($)
);


sub FHSBHS ($)
{


my $fileName= shift;
open(FILEREAD, "< $fileName");
$FILE = "$fileName";
open(FILE) or die("Could not open file.");
foreach $line (<FILE>)
{
my $hex = unpack( 'H*', $line );
my $segsep = "0a";
my $recsep ="0a 0b";
$hex =~ s/$segsep/$recsep /gi ;
print "$hex\n";
open(OUTPUT, "> $fileName") or die "Can't open output file $fileName\n";
print OUTPUT $hex;
close OUTPUT;
print "\n";
}
close FILEREAD;
}

#calling the above function with one command line parameter i.e filename.
FHSBHS($ARGV[0]);
################################################## ###########
I am having few problems with this script.
1.File is changed to hex format and the sequence seemed change but not able to convert back to ASCII.File is written in the same HEX format.

please help how it can be written back to the ASCII format with the changed HEX sequence .

[miller]

Hi my partner in crime somyasager. Here is your latest script attempt with comments added at lines you should think about. Your script is currently full of logic errors, so read my comments and maybe you'll learn something.

#!/usr/bin/perl -w
 
#Perl script to change the HEX sequence from a given value to new value.
#Here "0a" in replaced by "0a 0b"
 
package FooterHeader;
 
require Exporter;
our @ISA = qw(Exporter);
#ERROR: read up on Exporter.  This statement exports a sub named "sub", and something else I can't even guess at.  Only include the explicit function name in your export. ie: our @EXPORT = qw(FHSBHS);
our @EXPORT = qw(
sub FHSBHS($)
);
 
#WARNING: Add "use strict;" here.  It will help you avoid any errors.
 
sub FHSBHS ($)
{
    my $fileName= shift;
    open(FILEREAD, "< $fileName");
#WARNING: Why create a second input file handle here named FILE when you already have FILEREAD?  I'm guessing that this is a debugging step, but this is a bad idea.  Just use FILEREAD.
    $FILE = "$fileName";
    open(FILE) or die("Could not open file.");
    foreach $line (<FILE>)
    {
        my $hex = unpack( 'H*', $line );
        my $segsep = "0a";
#ERROR: Do you really want a space between these two characters?  I'm guessing that your copying the formatting that you see in your hex editor.  But this will screw up the reverse operation
        my $recsep ="0a 0b";
#ERROR: Do you really want a space after the replaced value?  I don't think  you do.  Remove the space after $recsep.
        $hex =~ s/$segsep/$recsep /gi ;
#ERROR: Great.  You've done the unpacking, the search and replace, but what about repacking teh string?.  It would look something (maybe exactly) like this: $line = pack( 'H*', $hex);
        print "$hex\n";
 
#ERROR: BAD BAD BAD.  You're attempting to open and write to a file that you are currently reading from.  This might work if you were working with fixed record length files, but you are explicitly increasing the number of characters in your line.  This will cause bad things.  You need to open an output stream to a different file, before starting your file read loop, and then simply move the newly created file over your old file at the end of hte transformations.
        open(OUTPUT, "> $fileName") or die "Can't open output file $fileName\n";
#WARNING: You're going to want to be printing the $line value and not $hex in the end I think.
        print OUTPUT $hex;
        close OUTPUT;
        print "\n";
    }
    close FILEREAD;
}
 
#calling the above function with one command line parameter i.e filename.
FHSBHS($ARGV[0]);