Problems with MS smart quotes -- NOTHING works

[cozzifantutti]

I've searched through these Macromedia forums, and just about everyplace else.
I've tried every script out there. I've downloaded the DeMoronizer UDF....to
no avail. My CF page is an action page invoked by an Adobe Acrobat form. The
form has a large text field into which users often paste text from Microsoft
Word. I am trying to replace the 'smart quotes' with quotes that we can then
save to an ntext field on SQL Server. Here is what the test text: Too many
people are using these 'smart quotes'. I wish they wouldn't. ...looks like as
it comes into the action page: Too many people are using these ???smart
quotes???. I wish they wouldn???t. I have tried the following code
suggestions, to no avail: 1) Demoronizer, invoked thusly: <cfinclude
template='DeMoronize.cfm'> <cfset MSText = '#form.testInput#'> <cfoutput>With
MS Latin-1 Extentions:<br>#MSText#</cfoutput> <cfset ValidText =
DeMoronize(MSText)> <cfoutput>Valid ASCII:<br>#ValidText#</cfoutput> 2)
<cfscript> function myfunction(text) { var i = 1; for (i = 145; i
LTE 148; i = i + 1) { Text = Replace(Text, Chr(i),
Chr(34), 'All'); } // Return Text; } </cfscript> 3)
Replace(Replace(mystring,chr(1?47),chr(34),'ALL'), chr(148),ch?r(34),'ALL') 4)
Putting: <cfcontent type='text/html; charset=utf-8'> <meta
http-equiv='Content-Type' content='text/html; charset=utf-8'> ...at the top of
the page Nothing works. Is it possible that there's something on the server
that needs to be set? How the heck can I get this to work, when all the
available scripts do nothing?

[Jim Wells]

cozzifantutti, you need to give up on trying to convert the smart quotes into
regular quotes. There are many other things Microsoft trys to be smart with
besides quotes. For example smart hyphens and i think some others. Your
solution is to play with unicode and different encoding types. I had this
problem a long time ago and I think the key here was to force the browser to
display the page in unicode. For example, in IE goto VIEW - ENCODING -
Unicode. I wish i remember exactly what I did, but maybe this is enough info
to guide you in the right direction. - Jim

[cozzifantutti]

;) Easier said than done, my friend. Telling the users to turn off smart
quotes, or telling them to change the quotes in the PDF file are not options,
so sayeth the powers that be. My mission.....quest.....thing [/Pippin] is to
find a way to make this work. I do have to clarify, however....the Replace
code DOES work when Word text with smart quotes is pasted into an ordinary HTML
textarea and then submitted to the action page. In theory, a PDF form works
exactly the same way when HTML export is invoked (which it is). However, it's
clear that Acrobat does something once the text is pasted in there, because
what gets output for ALL of these characters is garbage consisting of three
characters: 1) an i with an umlaut (Unicode 00BE); 2) an upside-down question
mark (Unicode 00BF), and a symbol for 1/2 (Unicode BD). Because it's three
separate characters, it makes it hard to find what to replace. Any ideas?

[PaulH]

your unicode codepoints are wrong, those chars are actually EF BB BF and that's
a BOM for utf-8. supplying that in the middle of a text stream is a big honking
no-no. and when it is found, it's supposed to be interpreted as "ZERO WIDTH
NON-BREAKING SPACE".

reading thru these postings, i've lost the thread of what's happening. can you
restate your problem & the steps needed to re-create it?

[cozzifantutti]

Thank you thank you thank you for your reply! I sure hope you can help me.
Here's the problem, restated: I have a PDF file with over 300 form fields that
I'm submitting to a Cold Fusion action page. There is a routine that we run
which parses long multi-line entries (stored as ntext fields in the SQL Server
database) which converts CRLFs to the '\r' notation that Acrobat can read, so
paragraphs show up properly on re-load. I've attached the code to this
message. It's a custom tag. The problem arises when the user pastes text that
contains 'smart quotes' into the PDF form field from Microsoft Word. If you
paste such text into a standard HTML textarea and submit, this script:
<cfscript> form.wordquotes = Replace(form.wordquotes, chr(8217), chr(39));
form.wordquotes = Replace(form.wordquotes, chr(8216), chr(39));
form.wordquotes = Replace(form.wordquotes, chr(8220), chr(34));
form.wordquotes = Replace(form.wordquotes, chr(8221), chr(34)); </cfscript>
....works like a charm. But for all that PDF form fields are supposedly handled
just like HTML fields, they're not...or at the very least, Acrobat does
something to those quotes so that they are no longer using Microsoft's
encoding. Grabbing the ascii value of the garbage output, which looks like
this: ??? ...for all four types (single left and single right; double left
and double right), I get 65533. So when the routine divides this by 16 and
gets 4096, I get an error in the listgetat() function because there are only 16
characters in the hex list, and 4096 is out of range. The most recent thing I
tried was to surround the output text with URLEncodedFormat(), and got this:
Too%20many%20%EF%BF%BDpeople%EF%BF%BD%20aren%EF%BF %BDt%20using%20their%20heads%3
B%20they%EF%BF%BDre%20using%20these%20%EF%BF%BDsma rt%20quotes%EF%BF%BD%2E So
I got the EF, BF, BD that you referred to...except that it's not distinguishing
among the various quote characters. This wouldn't be a problem if they were
always double quotes, but sometimes they're single. The folks at Adobe are
telling me that when they export, they get %8d %8e %8f and %90. So what can I
do to trap for these? Telling users not to use smart quotes is not an option.
Thank you...I hope you can help!

<CFPARAM NAME="attributes.input" DEFAULT="This is the string to convert">
<CFPARAM NAME="attributes.output" DEFAULT="output">
<CFSET hex2string_output = "">

<!--- ------------------------------------------------------------ --->
<!--- first step is to convert string to hex --->
<!--- ------------------------------------------------------------ --->
<CFSET hexlist="0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F">
<CFSET string2hex_output="">

<CFLOOP FROM ="1" TO="#len(attributes.input)#" INDEX="i">
<CFSET thisasc = asc(mid(attributes.input,i,1))>
<CFSET thishex =
"#listgetat(hexlist,fix(thisasc/16)+1)##listgetat(hexlist,fix(thisasc mod
16)+1)#">
<CFSET string2hex_output = "#string2hex_output##thishex#">
</CFLOOP>

<!--- ------------------------------------------------------------ --->
<!--- string is now in hex. Replace very occurranc of crlf (x'0D0A') with \r
--->
<!--- ------------------------------------------------------------ --->
<CFSET newHex = replace(string2hex_output,"0D0A","5C72","ALL")>

<!--- ------------------------------------------------------------ --->
<!--- now convert from hex back to ascii string and return to caller --->
<!--- ------------------------------------------------------------ --->
<CFLOOP FROM ="1" TO="#len(newHex)#" INDEX="i" STEP="2">
<!--- first convert each hex pair to decimal --->
<cfset thisdec = inputBaseN(mid(newHex,i,2),16)>

<!--- now convert that decimal value to string value --->
<cfset thischr = chr(thisdec)>

<!--- add append to output string --->
<CFSET hex2string_output = "#hex2string_output##thischr#">
</CFLOOP>

<CFSET "caller.#attributes.output#" = hex2string_output>

[jmj]

Wish I could give you a positive answer. I know your pain, I've been through it
also. Our solution came down to making the end users change their behavior (ohh
the horror of it all) It's taken a lot of training. They had to do one of three
things: 1. Type their comments in notepad or simpletext (mac users) 2. If they
used Word or Appleworks they had to take their document and paste it into
notepad/simpletext and run replace all to rid of the nasty stuff ('/'/whatever
the characters). 3. If they did not do number 2 and just pasted the info into
the textarea they had to go through and manually change each offending
character. If you find a more elegant solution please post it. Good luck. J