two other problem with UTF-8 persian

[AmirAbbas]

hello again

i know that this forum is not suitable for this question
i must ask this question in phpbb.com site but i think i can find
the answer of my question here

there is two other problem with UTF-8 persian
first i must say something about persian language
persian is very similar to arabic language

arabic has 28 characters and persian has all of those 28 characters + 4 other characters
there is a character that is in both of this language
the pronunciation of this character in both language is the same, the usage is the same, everthing is the same
but unfortunately there is two code of this character in unicode

that character is this

persian one
ی

arabic one
ي

this character pronunced "ye" and it sounds is "ee"
as you can see there is a little diferences between arabic and persian character

the arabic one has two dot under character
this deferences caused biggest SEO problem for all persian sites
i said that both of these characters are supported in utf-8
in persian countries (iran, tajikistan, part of armenia, great part of pakistan, afghanestan)
there is a lot of arabic windows
for example one of my users have arabic windows, he come to my site and make a topic.
all "ye" characters are typed in arabic shape and because of UTF-8 support of arabic "ye"
that user can send his post without problem
and after that someone like me with persian windows come and make another post
that contain persian "ye" and he can send his post without problem

now imagine,

the result of search is not accurate because for each word that contain character "ye"
there is two record in database (one for arabic and one for persian)

for example you want to search this word

آموزشی

this word pronunced "amuzeshi" and means "educational"

if you type the word with arabic "ye" you can see a list of topic that contain this word
but when you type it with persian "ye" you will see different result
unfortunately google, yahoo and MSN can't recognize with site is persian and with one is arabic
they are not aware about this problem
in keyword tools in yahoo and google site it this two address

adwords google keyword selector

yahoo selector tool

if you type word with diferrent shape of "ye" you will see different result
for example test this two word in yahoo keyword selector tool

persian word

آموزشی

arabic word

آموزشي

pay attention that both of this two word is persian but in one of them we used arabic"ye"

now my request

can we replace all arabic "ye" with persian "ye" in

1-topic title
2-text body
3-search box

??

for example one person want to make a post that contain arabic "ye"
system replace all arabic "ye" with persian one and after that save that post
in database

and second problem
in UTF-8 there is lot of other characters
some Saboteur users come to site and register with some special character
after that they start to send some bad post
system can not find they user name . how can i change the username field in registeration
form that only english character be acceptable ?

[dcz]

Well, I am not sure this is this big of a deal.

For sure results will be different, just like when using or not using accents in french (é or e for the same word, just changing pronunciation, but it's still readable without) as many user won't use accents while other will.

What I think in the end is that people posting one way will as well perform search the same way, and we want to be findable on the two spellings.

People writing in your forums with the Arabic characters are more than likely to as well search using the same spelling, and the same goes with the Persian case.

So I am not sure it would be this wise to filter just one case as obviously user will continue searching the way they always did.

All you could do about this would be content wise, posting good articles specifically about the keyword and spelling you want to SEO, one article or at list part for each spelling. And to post many links to them

And it's quite a good geofilter too, let me explain.

If one of you keyword/key-topic is using this 'ye', you can write down two articles about it, one using the Arabic 'ye' and the other one the Persians.
You'll be able to personalize a bit the two articles because you know the one using the Arabic 'ye' will most likely be found by people using the Arabic char-set

Now if the subject is really not personalize-able upon such criteria, then again, you should make sure to use the two spelling the most possible to be findable by your two types of users.

Because if you filter those, user with Arabic char-set won't find you as easy on those terms, as I bet there is more difference in search engine results in such cases than in the 'é' 'e' french accent case.

Then, talking about your registering problem, we could try using the UTF-8 filtering technique I put together.

Try open :

[AmirAbbas]

[dcz]

Good

I think the easiest to deal with this without to much changes would just be to edit the $lang['Username_taken'] key in lang_main.php, you can use html in those, thus a nice <br /> to tell that in addition, nicknames can only contain Latin characters for security reasons.

I am pretty sure a better filtering is possible here, like making sure the nickname does only contain Latin and / or one of the 32 allowed Persian characters, if you'd like to still allow Persian for nick names. Unfortunately, I don't know much about char-sets, even though I have been learning quite a lot thanks to you , later maybe ...


++

[AmirAbbas]

[dcz]

Yes and here it is

ی is

[AmirAbbas]

unfortunately it doesn't work

after i apllied this change and i tried to send a post
see this picture



the titles are missed
and there is no diferrence between english and persian in this case
if you use only english or only persian, in both case the title will miss

[dcz]

Sorry there was a typo in the replace, $url instead of $subject erf, try :

[AmirAbbas]

[dcz]

All right, let's try it just before the post is submitted then.

Open :

[AmirAbbas]

problem again

titles are missed

[dcz]

All right, I think I understood our problem.

The first suggested code works, but you need to enter the HTML ASCII codes instead of the copied characters from here.

eg :

Open :

[AmirAbbas]

i do it before

i qouted your message and i clicked on preview button
with this method you can see the source in correct way

it didn't work

[dcz]

mmmmh.

I have found out a simple, working in UTF-8, the bug came from the htmlspecialchars to break html ascii as it replaces all & with &amp; it breaks all &# And actually this would break any char not being part of the defined char-set and thus coded in html ascii.

So here is how I turned around :

Open :

[AmirAbbas]