Weird strings getting written over each other

[Bonj]

Hello. I hope somebody can help me on this, because I'm running out of
options to turn to.

I have almost solved my regular expression function. Basically it works OK
if unicode is defined. It doesn't work OK in ANSI mode however, as it has to
use MultiByteToWideChar and WideCharToMultiByte.
I've discovered that the regular expression part is working fine. As far as
I can tell the regular expression code is correctly parsing what it
receives, and outputting it in the correct format. However the calculation
of what it's going to receive is going wrong.
Basically, two weird things are happening which are baffling me:
1) When I run it with /Zi and /D"_DEBUG" defined, but *not* being debugged
under WinDbg, then the pattern gets expanded OK. The text gets expanded OK.
But when the text expands, it overwrites 'pattern' with the last word of it!
The output of the following is:
Pattern is expanded to "\w+"
Text is expanded to "A Collection of words"
But pattern is now "words"

bPattern is "words"
bText is "A Collection of words"

2) When it's run when it *is* being debugged under WinDbg, then the strings
get copied fine! No overwriting, and the output is:
Pattern is expanded to "\w+"
Text is expanded to "A Collection of words"
But pattern is now "\w+"

bPattern is "\w+"
bText is "A Collection of words"
Match found "A" at 0 for 1
Match found "Collection" at 2 for 10
Match found "of" at 13 for 2
Match found "words" at 16 for 5

which is exactly correct - the only thing being, when I press F10 to step
over any call to 'free' - it crashes! Can't further debug the program or
anything.

I'm worried about two things. One, why 'free' is crashing WinDbg, and not
the program on its own, but more worryingly, why is MultiByteToWideChar
overwriting my string's memory, when this behaviour is not reproducible in
WinDbg. I previously had it using mbstowcs_s and it did exactly the same
thing. I even checked the memory locations in WinDbg and they were near each
other but plenty far enough apart (40 bytes apart IIRC).

Can anyone suggest a resolution? It would be really good if somebody could
reproduce this.

This is the current code:
typedef void (CALLBACK* MatchEvaluator)(LPCTSTR value, long start, long
length, LPVOID extradata);
BOOL MatchWords(LPCTSTR pattern, LPCTSTR text, MatchEvaluator pfnMECB,
LPVOID extradata)
{
IRegExp* regexp = NULL;
IMatchCollection* pMatches = NULL;
IMatch* pMatch = NULL;
long nMatchCount, nMatch;
HRESULT hr;
BSTR bText, bPattern;
LPWSTR wtext, wpattern;
#ifdef _UNICODE
wtext = (LPWSTR)text;
wpattern = (LPWSTR)pattern;
#else
size_t textlen = _tcslen(text) + 1, patlen = _tcslen(pattern) + 1,
convtext, convpat;
wtext = (LPWSTR)malloc(textlen),
wpattern = (LPWSTR)malloc(patlen);

MultiByteToWideChar(CP_ACP, 0, (LPCSTR)pattern, (int)patlen, wpattern,
(int)patlen);
wpattern[patlen - 1] = L'\0';
wprintf(L"Pattern is expanded to \"%s\"\n", wpattern);

MultiByteToWideChar(CP_ACP, 0, (LPCSTR)text, (int)textlen, wtext,
(int)textlen); //THIS WRITES OVER pattern! why?
wtext[textlen - 1] = L'\0';
wprintf(L"Text is expanded to \"%s\"\n", wtext);
wprintf(L"But pattern is now \"%s\"\n", wpattern);

#endif

bText = SysAllocString((LPOLESTR)(wtext)),
bPattern = SysAllocString((LPOLESTR)(wpattern));

wprintf(L"\nbPattern is \"%s\"\n", bPattern);
wprintf(L"bText is \"%s\"\n", bText);

CoInitialize(NULL);
CoCreateInstance(&CLSID_RegExp, NULL, CLSCTX_INPROC_SERVER, &IID_IRegExp,
(void**)&regexp);
IRegExp_put_Global(regexp, VARIANT_TRUE);
IRegExp_put_IgnoreCase(regexp, VARIANT_TRUE);
IRegExp_put_Pattern(regexp, bPattern);
IRegExp_Execute(regexp, bText, (IDispatch**)&pMatches);
hr = IMatchCollection_get_Count(pMatches, &nMatchCount);
for(nMatch = 0; nMatch < nMatchCount; nMatch++)
{
BSTR bValue; LPWSTR sValue;
LPTSTR tValue;
long index, length;
size_t stringlen, converted;
IMatchCollection_get_Item(pMatches, nMatch, (IDispatch**)&pMatch);
IMatch_get_Value(pMatch, &bValue);
IMatch_get_FirstIndex(pMatch, &index);
IMatch_get_Length(pMatch, &length);
stringlen = 1 + SysStringLen(bValue);
sValue = (LPWSTR)malloc(stringlen);
swprintf_s(sValue, stringlen, L"%s", bValue);
sValue[stringlen - 1] = _T('\0');
#ifdef _UNICODE
tValue = sValue;
#else
tValue = (LPTSTR)malloc(stringlen);
WideCharToMultiByte(CP_ACP, 0, sValue, (int)stringlen, tValue,
(int)stringlen, NULL, NULL);
#endif
pfnMECB(tValue, index, length, extradata);
#ifndef _UNICODE
// #ifndef _DEBUG
// free(tValue); //crashes WinDbg - why?
// #endif
#endif
IMatch_Release(pMatch);
// #ifndef _DEBUG
// free(sValue);
// #endif
}

#ifndef _UNICODE
// #ifndef _DEBUG
// free(wtext);
// free(wpattern);
// #endif
#endif

SysFreeString(bText);
IMatchCollection_Release(pMatches);
IRegExp_Release(regexp);
CoUninitialize();
return TRUE;
}
void CALLBACK MatchEval(LPCTSTR value, long start, long length, LPVOID
extradata)
{
_tprintf(_T("Match found \"%s\" at %d for %d\n"), value, start, length);
}

int main()
{
MatchWords(_T("\\w+"), _T("A Collection of words"), MatchEval, NULL);
return 0;
}

[Bonj]

SORRY - Please ignore - this is SILLY question even for me...
i've only malloc'ed half the amount of bytes I actually require...

still weirded out about the free crashing windbg though...

"Bonj" <benjtaylor at hotpop d0t com> wrote in message
news:%2****************@TK2MSFTNGP09.phx.gbl...

Hello. I hope somebody can help me on this, because I'm running out of
options to turn to.

I have almost solved my regular expression function. Basically it works OK
if unicode is defined. It doesn't work OK in ANSI mode however, as it has
to use MultiByteToWideChar and WideCharToMultiByte.
I've discovered that the regular expression part is working fine. As far
as I can tell the regular expression code is correctly parsing what it
receives, and outputting it in the correct format. However the calculation
of what it's going to receive is going wrong.
Basically, two weird things are happening which are baffling me:
1) When I run it with /Zi and /D"_DEBUG" defined, but *not* being debugged
under WinDbg, then the pattern gets expanded OK. The text gets expanded
OK. But when the text expands, it overwrites 'pattern' with the last word
of it! The output of the following is:
Pattern is expanded to "\w+"
Text is expanded to "A Collection of words"
But pattern is now "words"

bPattern is "words"
bText is "A Collection of words"

2) When it's run when it *is* being debugged under WinDbg, then the
strings get copied fine! No overwriting, and the output is:
Pattern is expanded to "\w+"
Text is expanded to "A Collection of words"
But pattern is now "\w+"

bPattern is "\w+"
bText is "A Collection of words"
Match found "A" at 0 for 1
Match found "Collection" at 2 for 10
Match found "of" at 13 for 2
Match found "words" at 16 for 5

which is exactly correct - the only thing being, when I press F10 to step
over any call to 'free' - it crashes! Can't further debug the program or
anything.

I'm worried about two things. One, why 'free' is crashing WinDbg, and not
the program on its own, but more worryingly, why is MultiByteToWideChar
overwriting my string's memory, when this behaviour is not reproducible in
WinDbg. I previously had it using mbstowcs_s and it did exactly the same
thing. I even checked the memory locations in WinDbg and they were near
each other but plenty far enough apart (40 bytes apart IIRC).

Can anyone suggest a resolution? It would be really good if somebody could
reproduce this.

This is the current code:
typedef void (CALLBACK* MatchEvaluator)(LPCTSTR value, long start, long
length, LPVOID extradata);
BOOL MatchWords(LPCTSTR pattern, LPCTSTR text, MatchEvaluator pfnMECB,
LPVOID extradata)
{
IRegExp* regexp = NULL;
IMatchCollection* pMatches = NULL;
IMatch* pMatch = NULL;
long nMatchCount, nMatch;
HRESULT hr;
BSTR bText, bPattern;
LPWSTR wtext, wpattern;
#ifdef _UNICODE
wtext = (LPWSTR)text;
wpattern = (LPWSTR)pattern;
#else
size_t textlen = _tcslen(text) + 1, patlen = _tcslen(pattern) + 1,
convtext, convpat;
wtext = (LPWSTR)malloc(textlen),
wpattern = (LPWSTR)malloc(patlen);

MultiByteToWideChar(CP_ACP, 0, (LPCSTR)pattern, (int)patlen, wpattern,
(int)patlen);
wpattern[patlen - 1] = L'\0';
wprintf(L"Pattern is expanded to \"%s\"\n", wpattern);

MultiByteToWideChar(CP_ACP, 0, (LPCSTR)text, (int)textlen, wtext,
(int)textlen); //THIS WRITES OVER pattern! why?
wtext[textlen - 1] = L'\0';
wprintf(L"Text is expanded to \"%s\"\n", wtext);
wprintf(L"But pattern is now \"%s\"\n", wpattern);

#endif

bText = SysAllocString((LPOLESTR)(wtext)),
bPattern = SysAllocString((LPOLESTR)(wpattern));

wprintf(L"\nbPattern is \"%s\"\n", bPattern);
wprintf(L"bText is \"%s\"\n", bText);

CoInitialize(NULL);
CoCreateInstance(&CLSID_RegExp, NULL, CLSCTX_INPROC_SERVER, &IID_IRegExp,
(void**)&regexp);
IRegExp_put_Global(regexp, VARIANT_TRUE);
IRegExp_put_IgnoreCase(regexp, VARIANT_TRUE);
IRegExp_put_Pattern(regexp, bPattern);
IRegExp_Execute(regexp, bText, (IDispatch**)&pMatches);
hr = IMatchCollection_get_Count(pMatches, &nMatchCount);
for(nMatch = 0; nMatch < nMatchCount; nMatch++)
{
BSTR bValue; LPWSTR sValue;
LPTSTR tValue;
long index, length;
size_t stringlen, converted;
IMatchCollection_get_Item(pMatches, nMatch, (IDispatch**)&pMatch);
IMatch_get_Value(pMatch, &bValue);
IMatch_get_FirstIndex(pMatch, &index);
IMatch_get_Length(pMatch, &length);
stringlen = 1 + SysStringLen(bValue);
sValue = (LPWSTR)malloc(stringlen);
swprintf_s(sValue, stringlen, L"%s", bValue);
sValue[stringlen - 1] = _T('\0');
#ifdef _UNICODE
tValue = sValue;
#else
tValue = (LPTSTR)malloc(stringlen);
WideCharToMultiByte(CP_ACP, 0, sValue, (int)stringlen, tValue,
(int)stringlen, NULL, NULL);
#endif
pfnMECB(tValue, index, length, extradata);
#ifndef _UNICODE
// #ifndef _DEBUG
// free(tValue); //crashes WinDbg - why?
// #endif
#endif
IMatch_Release(pMatch);
// #ifndef _DEBUG
// free(sValue);
// #endif
}

#ifndef _UNICODE
// #ifndef _DEBUG
// free(wtext);
// free(wpattern);
// #endif
#endif

SysFreeString(bText);
IMatchCollection_Release(pMatches);
IRegExp_Release(regexp);
CoUninitialize();
return TRUE;
}
void CALLBACK MatchEval(LPCTSTR value, long start, long length, LPVOID
extradata)
{
_tprintf(_T("Match found \"%s\" at %d for %d\n"), value, start, length);
}

int main()
{
MatchWords(_T("\\w+"), _T("A Collection of words"), MatchEval, NULL);
return 0;
}

[Bonj]

still weirded out about the free crashing windbg though...

SAME ISSUE! works fine now... was trying to free two at once.

Although what IS weird is how it *works* fine in WinDbg.....even when I
don't malloc (stringlen * sizeof(wchar_t))...

"Bonj" <benjtaylor at hotpop d0t com> wrote in message
news:%2****************@TK2MSFTNGP09.phx.gbl...

Hello. I hope somebody can help me on this, because I'm running out of
options to turn to.

I have almost solved my regular expression function. Basically it works
OK if unicode is defined. It doesn't work OK in ANSI mode however, as it
has to use MultiByteToWideChar and WideCharToMultiByte.
I've discovered that the regular expression part is working fine. As far
as I can tell the regular expression code is correctly parsing what it
receives, and outputting it in the correct format. However the
calculation of what it's going to receive is going wrong.
Basically, two weird things are happening which are baffling me:
1) When I run it with /Zi and /D"_DEBUG" defined, but *not* being
debugged under WinDbg, then the pattern gets expanded OK. The text gets
expanded OK. But when the text expands, it overwrites 'pattern' with the
last word of it! The output of the following is:
Pattern is expanded to "\w+"
Text is expanded to "A Collection of words"
But pattern is now "words"

bPattern is "words"
bText is "A Collection of words"

2) When it's run when it *is* being debugged under WinDbg, then the
strings get copied fine! No overwriting, and the output is:
Pattern is expanded to "\w+"
Text is expanded to "A Collection of words"
But pattern is now "\w+"

bPattern is "\w+"
bText is "A Collection of words"
Match found "A" at 0 for 1
Match found "Collection" at 2 for 10
Match found "of" at 13 for 2
Match found "words" at 16 for 5

which is exactly correct - the only thing being, when I press F10 to step
over any call to 'free' - it crashes! Can't further debug the program or
anything.

I'm worried about two things. One, why 'free' is crashing WinDbg, and not
the program on its own, but more worryingly, why is MultiByteToWideChar
overwriting my string's memory, when this behaviour is not reproducible
in WinDbg. I previously had it using mbstowcs_s and it did exactly the
same thing. I even checked the memory locations in WinDbg and they were
near each other but plenty far enough apart (40 bytes apart IIRC).

Can anyone suggest a resolution? It would be really good if somebody
could reproduce this.

This is the current code:
typedef void (CALLBACK* MatchEvaluator)(LPCTSTR value, long start, long
length, LPVOID extradata);
BOOL MatchWords(LPCTSTR pattern, LPCTSTR text, MatchEvaluator pfnMECB,
LPVOID extradata)
{
IRegExp* regexp = NULL;
IMatchCollection* pMatches = NULL;
IMatch* pMatch = NULL;
long nMatchCount, nMatch;
HRESULT hr;
BSTR bText, bPattern;
LPWSTR wtext, wpattern;
#ifdef _UNICODE
wtext = (LPWSTR)text;
wpattern = (LPWSTR)pattern;
#else
size_t textlen = _tcslen(text) + 1, patlen = _tcslen(pattern) + 1,
convtext, convpat;
wtext = (LPWSTR)malloc(textlen),
wpattern = (LPWSTR)malloc(patlen);

MultiByteToWideChar(CP_ACP, 0, (LPCSTR)pattern, (int)patlen, wpattern,
(int)patlen);
wpattern[patlen - 1] = L'\0';
wprintf(L"Pattern is expanded to \"%s\"\n", wpattern);

MultiByteToWideChar(CP_ACP, 0, (LPCSTR)text, (int)textlen, wtext,
(int)textlen); //THIS WRITES OVER pattern! why?
wtext[textlen - 1] = L'\0';
wprintf(L"Text is expanded to \"%s\"\n", wtext);
wprintf(L"But pattern is now \"%s\"\n", wpattern);

#endif

bText = SysAllocString((LPOLESTR)(wtext)),
bPattern = SysAllocString((LPOLESTR)(wpattern));

wprintf(L"\nbPattern is \"%s\"\n", bPattern);
wprintf(L"bText is \"%s\"\n", bText);

CoInitialize(NULL);
CoCreateInstance(&CLSID_RegExp, NULL, CLSCTX_INPROC_SERVER,
&IID_IRegExp, (void**)&regexp);
IRegExp_put_Global(regexp, VARIANT_TRUE);
IRegExp_put_IgnoreCase(regexp, VARIANT_TRUE);
IRegExp_put_Pattern(regexp, bPattern);
IRegExp_Execute(regexp, bText, (IDispatch**)&pMatches);
hr = IMatchCollection_get_Count(pMatches, &nMatchCount);
for(nMatch = 0; nMatch < nMatchCount; nMatch++)
{
BSTR bValue; LPWSTR sValue;
LPTSTR tValue;
long index, length;
size_t stringlen, converted;
IMatchCollection_get_Item(pMatches, nMatch, (IDispatch**)&pMatch);
IMatch_get_Value(pMatch, &bValue);
IMatch_get_FirstIndex(pMatch, &index);
IMatch_get_Length(pMatch, &length);
stringlen = 1 + SysStringLen(bValue);
sValue = (LPWSTR)malloc(stringlen);
swprintf_s(sValue, stringlen, L"%s", bValue);
sValue[stringlen - 1] = _T('\0');
#ifdef _UNICODE
tValue = sValue;
#else
tValue = (LPTSTR)malloc(stringlen);
WideCharToMultiByte(CP_ACP, 0, sValue, (int)stringlen, tValue,
(int)stringlen, NULL, NULL);
#endif
pfnMECB(tValue, index, length, extradata);
#ifndef _UNICODE
// #ifndef _DEBUG
// free(tValue); //crashes WinDbg - why?
// #endif
#endif
IMatch_Release(pMatch);
// #ifndef _DEBUG
// free(sValue);
// #endif
}

#ifndef _UNICODE
// #ifndef _DEBUG
// free(wtext);
// free(wpattern);
// #endif
#endif

SysFreeString(bText);
IMatchCollection_Release(pMatches);
IRegExp_Release(regexp);
CoUninitialize();
return TRUE;
}
void CALLBACK MatchEval(LPCTSTR value, long start, long length, LPVOID
extradata)
{
_tprintf(_T("Match found \"%s\" at %d for %d\n"), value, start, length);
}

int main()
{
MatchWords(_T("\\w+"), _T("A Collection of words"), MatchEval, NULL);
return 0;
}

[Steve Friedl [MVP/Security]]

"Bonj" <benjtaylor at hotpop d0t com> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...

Although what IS weird is how it *works* fine in WinDbg.....even when I
don't malloc (stringlen * sizeof(wchar_t))...

That's not enough space - no room for the type - and you should not be
naming the type (wchar_t) explicitly. You could say sizeof(TCHAR), or (even
better) extract the size from the pointed-to object:

TCHAR *p = malloc( (stringlen + 1) * sizeof(*p) );

Steve

[Steve Friedl [MVP/Security]]

"Steve Friedl [MVP/Security]" <st***@unixwiz.net> wrote in message
news:uZ**************@TK2MSFTNGP10.phx.gbl...

That's not enough space - no room for the type

Which should be "no room for the NUL"...

Steve
--
Steve Friedl -- Tustin, California USA -- www.unixwiz.net
Unix Wizard -- Microsoft MVP/Security -- I speak for me only

[Bonj]

Steve
Normally I wouldn't name the type explicity and always use _TCHAR*. However
in this instance, the string HAS to be converted to a unicode string - so it
can be converted to a BSTR.
If you do
char* thestring = "Hello";
BSTR bStr = ((LPOLESTR)thestring);
it won't convert properly.

But if you do
wchar_t* thestring = L"hello";
BSTR bStr = ((LPOLESTR)thestring);
it seems to be OK.

Presumably a BSTR is always unicode?
"Steve Friedl [MVP/Security]" wrote:

"Bonj" <benjtaylor at hotpop d0t com> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...

Although what IS weird is how it *works* fine in WinDbg.....even when I
don't malloc (stringlen * sizeof(wchar_t))...

That's not enough space - no room for the type - and you should not be
naming the type (wchar_t) explicitly. You could say sizeof(TCHAR), or (even
better) extract the size from the pointed-to object:

TCHAR *p = malloc( (stringlen + 1) * sizeof(*p) );

Steve

[James Curran]

"Bonj" <benjtaylor at hotpop d0t com> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...

Although what IS weird is how it *works* fine in WinDbg.....even when I
don't malloc (stringlen * sizeof(wchar_t))...

The debugger put things on the stack which, when overwritten, aren't
fatal.

In years of my own coding & answering questions on newsgroups, I've
found in 100% of the cases where a program works in the debugger & fails
outside of it, the bounds of an array were exceeded.

--
Truth,
James Curran
[erstwhile VC++ MVP]
Home: www.noveltheory.com Work: www.njtheater.com
Blog: www.honestillusion.com Day Job: www.partsearch.com

[Steve Friedl [MVP/Security]]

"James Curran" <Ja*********@mvps.org> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...

In years of my own coding & answering questions on newsgroups, I've
found in 100% of the cases where a program works in the debugger & fails
outside of it, the bounds of an array were exceeded.

Geek jargon: a program error that fails to manifest itself when run under
the debugger is known as a "Heisenbug" :-)

Steve
--
Steve Friedl -- Tustin, California USA -- www.unixwiz.net
Unix Wizard -- Microsoft MVP/Security -- I speak for me only

[Bonj]

Although what IS weird is how it *works* fine in WinDbg.....even when I
don't malloc (stringlen * sizeof(wchar_t))...
The debugger put things on the stack

Even as a result of malloc? Why would it do that, and the 'normal' program
uses the heap? Is it because of windbg, or that the program's compiled with
debug options (/Zi /D"_DEBUG") ?

which, when overwritten, aren't
fatal.


In years of my own coding & answering questions on newsgroups, I've
found in 100% of the cases where a program works in the debugger & fails
outside of it, the bounds of an array were exceeded.

--
Truth,
James Curran
[erstwhile VC++ MVP]
Home: www.noveltheory.com Work: www.njtheater.com
Blog: www.honestillusion.com Day Job: www.partsearch.com

[James Curran]

"Bonj" <benjtaylor at hotpop d0t com> wrote in message
news:uR**************@TK2MSFTNGP10.phx.gbl...

Even as a result of malloc? Why would it do that, and the 'normal' program
uses the heap? Is it because of windbg, or that the program's compiled with debug options (/Zi /D"_DEBUG") ?

It doesn't put your stuff on the stack, it puts it's own stuff there.
Basically, when running on the debugger, you have two different programs
running together, and sharing the stack.

--
Truth,
James Curran
[erstwhile VC++ MVP]
Home: www.noveltheory.com Work: www.njtheater.com
Blog: www.honestillusion.com Day Job: www.partsearch.com

[Jerry Coffin]

In article <#X**************@TK2MSFTNGP14.phx.gbl>,
Ja*********@mvps.org says...

[ ... ]

Basically, when running on the debugger, you have two different programs
running together, and sharing the stack.

The debugger has its own stack, completely separate from the
debugee's stack.

--
Later,
Jerry.

The universe is a figment of its own imagination.