Win32 Application CryptoAPI

[Darren Bennett]

Hi There,

I have been scanning the newsgroups for a solution to my problem and have
found that a few others are also experiencing the same problem but none of
the solutions provided to them seem to work for me.

I have a native Win32 application (written in C++) that needs to encrypt
some data, pass that data to a web service (written in C#) for decryption. I
just can't seem to decrypt the data correctly.

To demonstrate the problem, I have written a native Win32 (C++) application
to encrypt some data and write the encrypted data out to disk:

HCRYPTPROV hProv;
HCRYPTHASH hHash;
HCRYPTKEY hKey;
HANDLE hFile;
DWORD dwNumBytes, dwBytesWritten;
BYTE byPassword[] = {65, 66, 67, 68, 69, 70}; // equals "ABCDEF"
BYTE byData[] = {10, 20, 30, 40, 50, 60, 70, 80, 0, 0, 0, 0, 0, 0, 0,
0};

// Create the Cryptograhic Provider Object
if (CryptAcquireContext(&hProv, NULL, MS_DEF_PROV, PROV_RSA_FULL,
CRYPT_VERIFYCONTEXT)) {
// Create a hash object
if (CryptCreateHash(hProv, CALG_MD5, 0, 0, &hHash)) {
// Hash in the password
if (CryptHashData(hHash, byPassword, sizeof(byPassword), 0)) {
// Derive a session key from the hash object.
if (CryptDeriveKey(hProv, CALG_RC2, hHash, 0, &hKey)) {
dwNumBytes = 8;
if (CryptEncrypt(hKey, NULL, TRUE, 0, byData, &dwNumBytes,
sizeof(byData))) {
// Create a file to store the encrypted data
if ((hFile = CreateFile(_T("C:\\Encrypted.dat"), GENERIC_WRITE, 0,
NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL)) != INVALID_HANDLE_VALUE) {
// Write the encrypted data out to file
WriteFile(hFile, &byData, dwNumBytes, &dwBytesWritten, NULL);
// Close the file handdle
CloseHandle(hFile);
}
}
// Destroy the session key
CryptDestroyKey(hKey);
}
}
// Destroy the hash object
CryptDestroyHash(hHash);
}
// Release the cryptographic provider context
CryptReleaseContext(hProv, 0);
}

I have also written a .NET C# application to read the data from file and try
and decrypt the data:

int iNumBytes;
FileStream fsData;
byte[] byData = new byte[16];
byte[] bySalt = {0,0,0,0,0,0,0,0};
byte[] byInitVect = {0,0,0,0,0,0,0,0};

// Open the file containing the encrypted data, read inb the data and close
the file
fsData = File.Open("C:\\Encrypted.dat", FileMode.Open);
iNumBytes = fsData.Read(byData, 0, byData.Length);
fsData.Close();

// Derive a session key from the password using an MD5 hash
PasswordDeriveBytes SessionKey = new PasswordDeriveBytes("ABCDEF", bySalt,
"MD5", 1);

// Set up an RC2 cryptographic object
RC2CryptoServiceProvider Rc2 = new RC2CryptoServiceProvider();
Rc2.Mode = CipherMode.CBC;
Rc2.KeySize = 40;
Rc2.EffectiveKeySize = 40;
Rc2.BlockSize = 64;
Rc2.Key = SessionKey.GetBytes(5);

// Decrypt the data
ICryptoTransform decryptor = Rc2.CreateDecryptor(SessionKey.GetBytes(8),
byInitVect);
byte[] myOutputBytes = new byte[decryptor.OutputBlockSize];
iNumBytes = decryptor.TransformBlock(byData, 0, decryptor.InputBlockSize,
myOutputBytes, 0);

The problem is that the data just does not decrypt. I really don't care what
type of hashing or encryption algorithms are used so long as they are
supported on Win98/Me and NT4.0SP6/2000/XP/2003.

Any help in solving this problem would be greatly appreciated.

Thanks,

-Darren-

[Doug Barlow]

Darren,

I haven't tried running your programs yet, but I do have a few suggestions.

First, you've probably chosen the worst case algorithm for compatibility --
The old Microsoft CSPs mess with 40-bit RC2 to try to make it stronger,
giving it an 88-byte salt, so you've got to manage the salt values, too.
You might have better luck moving up to a 128-bit RC2 key, which doesn't
default to a special salt value. Take a look at the MSDN documentation for
CryptDeriveKey for details.

Next, I wouldn't assume the .NET PasswordDeriveBytes class derives the same
key as the Win32 CryptDeriveKey service. In fact, I'd bet against it. You
should come up with a common method for deriving a key, and make sure you
use the same key and salt in both applications. You can import a plain-text
session key into CryptoAPI by following the example shown in
[url]http://support.microsoft.com/default.aspx?scid=kb;en-us;228786[/url].

I hope that helps,

Doug Barlow
The Soft Pedal Shop
[url]http://www.softpedal.net[/url]


"Darren Bennett" <darren@work.com> wrote in message
news:167A699B-452A-4574-8FE4-00AE6ACBA4DC@microsoft.com...

> Hi There,
>
> I have been scanning the newsgroups for a solution to my problem and have
> found that a few others are also experiencing the same problem but none of
> the solutions provided to them seem to work for me.
> ...

[Darren Bennett]

Hi Doug,

Thanks for your reply to my question.

I have modified the code to make it 128bit RC2 encryption as you suggested
but I still end up with the same result. I have included the updated code
below:

Win32 Application:
============
HCRYPTPROV hProv;
HCRYPTHASH hHash;
HCRYPTKEY hKey;
HANDLE hFile;
DWORD dwNumBytes, dwBytesWritten;
BYTE byPassword[] = {65, 66, 67, 68, 69, 70}; // equals "ABCDEF"
BYTE byData[] = {10, 20, 30, 40, 50, 60, 70, 80, 0, 0, 0, 0, 0, 0,
0, 0};

// Create the Cryptograhic Provider Object
if (CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL,
CRYPT_VERIFYCONTEXT)) {
// Create a hash object
if (CryptCreateHash(hProv, CALG_MD5, 0, 0, &hHash)) {
// Hash in the password
if (CryptHashData(hHash, byPassword, sizeof(byPassword), 0)) {
// Derive a session key from the hash object.
if (CryptDeriveKey(hProv, CALG_RC2, hHash, CRYPT_NO_SALT, &hKey)) {
dwNumBytes = 8;
if (CryptEncrypt(hKey, NULL, TRUE, 0, byData, &dwNumBytes,
sizeof(byData))) {
// Create a file to store the encrypted data
if ((hFile = CreateFile(_T("C:\\Encrypted.dat"), GENERIC_WRITE,
0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL)) !=
INVALID_HANDLE_VALUE) {
// Write the encrypted data out to file
WriteFile(hFile, &byData, dwNumBytes, &dwBytesWritten, NULL);
// Close the file handdle
CloseHandle(hFile);
}
}
// Destroy the session key
CryptDestroyKey(hKey);
}
}
// Destroy the hash object
CryptDestroyHash(hHash);
}
// Release the cryptographic provider context
CryptReleaseContext(hProv, 0);
}

C# Application:
==========
int iNumBytes;
FileStream fsData;
byte[] byData = new byte[16];
byte[] byInitVect = {0,0,0,0,0,0,0,0};

// Open the file containing the encrypted data, read inb the data and close
the file
fsData = File.Open("C:\\Encrypted.dat", FileMode.Open);
iNumBytes = fsData.Read(byData, 0, byData.Length);
fsData.Close();

// Derive a session key from the password using an MD5 hash
PasswordDeriveBytes SessionKey = new PasswordDeriveBytes("ABCDEF", null,
"MD5", 1);

// Set up an RC2 cryptographic object
RC2CryptoServiceProvider Rc2 = new RC2CryptoServiceProvider();
Rc2.Mode = CipherMode.CBC;
Rc2.KeySize = 128;
Rc2.EffectiveKeySize = 128;
Rc2.BlockSize = 64;

// Decrypt the data
ICryptoTransform decryptor = Rc2.CreateDecryptor(SessionKey.GetBytes(16),
byInitVect);
byte[] myOutputBytes = new byte[decryptor.OutputBlockSize];
iNumBytes = decryptor.TransformBlock(byData, 0, decryptor.InputBlockSize,
myOutputBytes, 0);


Any help would be greatly appreciated.

Thanks,

- Darren -



"Doug Barlow" wrote:

> Darren,
>
> I haven't tried running your programs yet, but I do have a few suggestions.
>
> First, you've probably chosen the worst case algorithm for compatibility --
> The old Microsoft CSPs mess with 40-bit RC2 to try to make it stronger,
> giving it an 88-byte salt, so you've got to manage the salt values, too.
> You might have better luck moving up to a 128-bit RC2 key, which doesn't
> default to a special salt value. Take a look at the MSDN documentation for
> CryptDeriveKey for details.
>
> Next, I wouldn't assume the .NET PasswordDeriveBytes class derives the same
> key as the Win32 CryptDeriveKey service. In fact, I'd bet against it. You
> should come up with a common method for deriving a key, and make sure you
> use the same key and salt in both applications. You can import a plain-text
> session key into CryptoAPI by following the example shown in
> [url]http://support.microsoft.com/default.aspx?scid=kb;en-us;228786[/url].
>
> I hope that helps,
>
> Doug Barlow
> The Soft Pedal Shop
> [url]http://www.softpedal.net[/url]
>
>
> "Darren Bennett" <darren@work.com> wrote in message
> news:167A699B-452A-4574-8FE4-00AE6ACBA4DC@microsoft.com...

> > Hi There,
> >
> > I have been scanning the newsgroups for a solution to my problem and have
> > found that a few others are also experiencing the same problem but none of
> > the solutions provided to them seem to work for me.
> > ...

>
>
>

[Joe Kaplan \(MVP - ADSI\)]

You are still making the assumption that PasswordDeriveBytes is going to
yield the same key as your CryptCreateHash/CryptHashData thing is going to
return the same key. Even if they use the same routine under the hood, you
have to remember that strings in .NET are Unicode and your unmanged routine
uses ASCII, so those shouldn't produce the same data.

You might start by hard-coding a key and going from there.

Joe K.

"Darren Bennett" <darren@work.com> wrote in message
news:295D64BB-051C-46CE-86F9-D1AD02FFA43D@microsoft.com...

> Hi Doug,
>
> Thanks for your reply to my question.
>
> I have modified the code to make it 128bit RC2 encryption as you suggested
> but I still end up with the same result. I have included the updated code
> below:
>
> Win32 Application:
> ============
> HCRYPTPROV hProv;
> HCRYPTHASH hHash;
> HCRYPTKEY hKey;
> HANDLE hFile;
> DWORD dwNumBytes, dwBytesWritten;
> BYTE byPassword[] = {65, 66, 67, 68, 69, 70}; // equals "ABCDEF"
> BYTE byData[] = {10, 20, 30, 40, 50, 60, 70, 80, 0, 0, 0, 0, 0, 0,
> 0, 0};
>
> // Create the Cryptograhic Provider Object
> if (CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL,
> CRYPT_VERIFYCONTEXT)) {
> // Create a hash object
> if (CryptCreateHash(hProv, CALG_MD5, 0, 0, &hHash)) {
> // Hash in the password
> if (CryptHashData(hHash, byPassword, sizeof(byPassword), 0)) {
> // Derive a session key from the hash object.
> if (CryptDeriveKey(hProv, CALG_RC2, hHash, CRYPT_NO_SALT, &hKey)) {
> dwNumBytes = 8;
> if (CryptEncrypt(hKey, NULL, TRUE, 0, byData, &dwNumBytes,
> sizeof(byData))) {
> // Create a file to store the encrypted data
> if ((hFile = CreateFile(_T("C:\\Encrypted.dat"), GENERIC_WRITE,
> 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL)) !=
> INVALID_HANDLE_VALUE) {
> // Write the encrypted data out to file
> WriteFile(hFile, &byData, dwNumBytes, &dwBytesWritten, NULL);
> // Close the file handdle
> CloseHandle(hFile);
> }
> }
> // Destroy the session key
> CryptDestroyKey(hKey);
> }
> }
> // Destroy the hash object
> CryptDestroyHash(hHash);
> }
> // Release the cryptographic provider context
> CryptReleaseContext(hProv, 0);
> }
>
> C# Application:
> ==========
> int iNumBytes;
> FileStream fsData;
> byte[] byData = new byte[16];
> byte[] byInitVect = {0,0,0,0,0,0,0,0};
>
> // Open the file containing the encrypted data, read inb the data and
> close
> the file
> fsData = File.Open("C:\\Encrypted.dat", FileMode.Open);
> iNumBytes = fsData.Read(byData, 0, byData.Length);
> fsData.Close();
>
> // Derive a session key from the password using an MD5 hash
> PasswordDeriveBytes SessionKey = new PasswordDeriveBytes("ABCDEF", null,
> "MD5", 1);
>
> // Set up an RC2 cryptographic object
> RC2CryptoServiceProvider Rc2 = new RC2CryptoServiceProvider();
> Rc2.Mode = CipherMode.CBC;
> Rc2.KeySize = 128;
> Rc2.EffectiveKeySize = 128;
> Rc2.BlockSize = 64;
>
> // Decrypt the data
> ICryptoTransform decryptor = Rc2.CreateDecryptor(SessionKey.GetBytes(16),
> byInitVect);
> byte[] myOutputBytes = new byte[decryptor.OutputBlockSize];
> iNumBytes = decryptor.TransformBlock(byData, 0, decryptor.InputBlockSize,
> myOutputBytes, 0);
>
>
> Any help would be greatly appreciated.
>
> Thanks,
>
> - Darren -
>
>
>
> "Doug Barlow" wrote:
>

>> Darren,
>>
>> I haven't tried running your programs yet, but I do have a few
>> suggestions.
>>
>> First, you've probably chosen the worst case algorithm for
>> compatibility --
>> The old Microsoft CSPs mess with 40-bit RC2 to try to make it stronger,
>> giving it an 88-byte salt, so you've got to manage the salt values, too.
>> You might have better luck moving up to a 128-bit RC2 key, which doesn't
>> default to a special salt value. Take a look at the MSDN documentation
>> for
>> CryptDeriveKey for details.
>>
>> Next, I wouldn't assume the .NET PasswordDeriveBytes class derives the
>> same
>> key as the Win32 CryptDeriveKey service. In fact, I'd bet against it.
>> You
>> should come up with a common method for deriving a key, and make sure you
>> use the same key and salt in both applications. You can import a
>> plain-text
>> session key into CryptoAPI by following the example shown in
>> [url]http://support.microsoft.com/default.aspx?scid=kb;en-us;228786[/url].
>>
>> I hope that helps,
>>
>> Doug Barlow
>> The Soft Pedal Shop
>> [url]http://www.softpedal.net[/url]
>>
>>
>> "Darren Bennett" <darren@work.com> wrote in message
>> news:167A699B-452A-4574-8FE4-00AE6ACBA4DC@microsoft.com...

>> > Hi There,
>> >
>> > I have been scanning the newsgroups for a solution to my problem and
>> > have
>> > found that a few others are also experiencing the same problem but none
>> > of
>> > the solutions provided to them seem to work for me.
>> > ...

>>
>>
>>

[Darren Bennett]

Thanks for your feedback.

I just couldn't get this working regardless of what I tried, so I have
returned to good old Win32 and P/Invoked the required Win32 Crypto functions
and it works like a treat. It's a shame because I really wanted to get this
running in pure .NET environment.

Thanks everyone for your feedback on this problem.

- Darren -

"Joe Kaplan (MVP - ADSI)" wrote:

> You are still making the assumption that PasswordDeriveBytes is going to
> yield the same key as your CryptCreateHash/CryptHashData thing is going to
> return the same key. Even if they use the same routine under the hood, you
> have to remember that strings in .NET are Unicode and your unmanged routine
> uses ASCII, so those shouldn't produce the same data.
>
> You might start by hard-coding a key and going from there.
>
> Joe K.
>
> "Darren Bennett" <darren@work.com> wrote in message
> news:295D64BB-051C-46CE-86F9-D1AD02FFA43D@microsoft.com...

> > Hi Doug,
> >
> > Thanks for your reply to my question.
> >
> > I have modified the code to make it 128bit RC2 encryption as you suggested
> > but I still end up with the same result. I have included the updated code
> > below:
> >
> > Win32 Application:
> > ============
> > HCRYPTPROV hProv;
> > HCRYPTHASH hHash;
> > HCRYPTKEY hKey;
> > HANDLE hFile;
> > DWORD dwNumBytes, dwBytesWritten;
> > BYTE byPassword[] = {65, 66, 67, 68, 69, 70}; // equals "ABCDEF"
> > BYTE byData[] = {10, 20, 30, 40, 50, 60, 70, 80, 0, 0, 0, 0, 0, 0,
> > 0, 0};
> >
> > // Create the Cryptograhic Provider Object
> > if (CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL,
> > CRYPT_VERIFYCONTEXT)) {
> > // Create a hash object
> > if (CryptCreateHash(hProv, CALG_MD5, 0, 0, &hHash)) {
> > // Hash in the password
> > if (CryptHashData(hHash, byPassword, sizeof(byPassword), 0)) {
> > // Derive a session key from the hash object.
> > if (CryptDeriveKey(hProv, CALG_RC2, hHash, CRYPT_NO_SALT, &hKey)) {
> > dwNumBytes = 8;
> > if (CryptEncrypt(hKey, NULL, TRUE, 0, byData, &dwNumBytes,
> > sizeof(byData))) {
> > // Create a file to store the encrypted data
> > if ((hFile = CreateFile(_T("C:\\Encrypted.dat"), GENERIC_WRITE,
> > 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL)) !=
> > INVALID_HANDLE_VALUE) {
> > // Write the encrypted data out to file
> > WriteFile(hFile, &byData, dwNumBytes, &dwBytesWritten, NULL);
> > // Close the file handdle
> > CloseHandle(hFile);
> > }
> > }
> > // Destroy the session key
> > CryptDestroyKey(hKey);
> > }
> > }
> > // Destroy the hash object
> > CryptDestroyHash(hHash);
> > }
> > // Release the cryptographic provider context
> > CryptReleaseContext(hProv, 0);
> > }
> >
> > C# Application:
> > ==========
> > int iNumBytes;
> > FileStream fsData;
> > byte[] byData = new byte[16];
> > byte[] byInitVect = {0,0,0,0,0,0,0,0};
> >
> > // Open the file containing the encrypted data, read inb the data and
> > close
> > the file
> > fsData = File.Open("C:\\Encrypted.dat", FileMode.Open);
> > iNumBytes = fsData.Read(byData, 0, byData.Length);
> > fsData.Close();
> >
> > // Derive a session key from the password using an MD5 hash
> > PasswordDeriveBytes SessionKey = new PasswordDeriveBytes("ABCDEF", null,
> > "MD5", 1);
> >
> > // Set up an RC2 cryptographic object
> > RC2CryptoServiceProvider Rc2 = new RC2CryptoServiceProvider();
> > Rc2.Mode = CipherMode.CBC;
> > Rc2.KeySize = 128;
> > Rc2.EffectiveKeySize = 128;
> > Rc2.BlockSize = 64;
> >
> > // Decrypt the data
> > ICryptoTransform decryptor = Rc2.CreateDecryptor(SessionKey.GetBytes(16),
> > byInitVect);
> > byte[] myOutputBytes = new byte[decryptor.OutputBlockSize];
> > iNumBytes = decryptor.TransformBlock(byData, 0, decryptor.InputBlockSize,
> > myOutputBytes, 0);
> >
> >
> > Any help would be greatly appreciated.
> >
> > Thanks,
> >
> > - Darren -
> >
> >
> >
> > "Doug Barlow" wrote:
> >

> >> Darren,
> >>
> >> I haven't tried running your programs yet, but I do have a few
> >> suggestions.
> >>
> >> First, you've probably chosen the worst case algorithm for
> >> compatibility --
> >> The old Microsoft CSPs mess with 40-bit RC2 to try to make it stronger,
> >> giving it an 88-byte salt, so you've got to manage the salt values, too.
> >> You might have better luck moving up to a 128-bit RC2 key, which doesn't
> >> default to a special salt value. Take a look at the MSDN documentation
> >> for
> >> CryptDeriveKey for details.
> >>
> >> Next, I wouldn't assume the .NET PasswordDeriveBytes class derives the
> >> same
> >> key as the Win32 CryptDeriveKey service. In fact, I'd bet against it.
> >> You
> >> should come up with a common method for deriving a key, and make sure you
> >> use the same key and salt in both applications. You can import a
> >> plain-text
> >> session key into CryptoAPI by following the example shown in
> >> [url]http://support.microsoft.com/default.aspx?scid=kb;en-us;228786[/url].
> >>
> >> I hope that helps,
> >>
> >> Doug Barlow
> >> The Soft Pedal Shop
> >> [url]http://www.softpedal.net[/url]
> >>
> >>
> >> "Darren Bennett" <darren@work.com> wrote in message
> >> news:167A699B-452A-4574-8FE4-00AE6ACBA4DC@microsoft.com...
> >> > Hi There,
> >> >
> >> > I have been scanning the newsgroups for a solution to my problem and
> >> > have
> >> > found that a few others are also experiencing the same problem but none
> >> > of
> >> > the solutions provided to them seem to work for me.
> >> > ...
> >>
> >>
> >>

>
>
>

[Pradeep Gupta via DotNetMonster.com]

Your problem lies in the way you are calling CryptDeriveKey.
You are not spicifying 40 anywhere. Try this:
CryptDeriveKey(hProv, CALG_RC2, hHashKey, (40 << 16) | CRYPT_NO_SALT, &hKey)) // key size in the upper 16 bits

Hope this helps,
Pradeep

--
Message posted via [url]http://www.dotnetmonster.com[/url]

[Darren Bennett]

Thanks for your feedback.

I tried specifing (128 << 16) for the 128 bit session key version of the
test code but it still didn't help. My feeling is that there is something
wrong with the padding because Win32 encryption uses "PKCS #1 Type 2 padding"
and .NET encryption does not support this and if I set the PaddingMode.Zeros
in the C# application actually decrypts something but the dat is incorrect.

I have been able to get around this incompatibility problem using Win32
Crypto API via P/Invkoe.

Thanks.

Cheers,

-Darren-

"Pradeep Gupta via DotNetMonster.com" wrote:

> Your problem lies in the way you are calling CryptDeriveKey.
> You are not spicifying 40 anywhere. Try this:
> CryptDeriveKey(hProv, CALG_RC2, hHashKey, (40 << 16) | CRYPT_NO_SALT, &hKey)) // key size in the upper 16 bits
>
> Hope this helps,
> Pradeep
>
> --
> Message posted via [url]http://www.dotnetmonster.com[/url]
>