Unicode standards, and how politics influence technology

[Cyborg Girl]

Hey Sigma, you should like this one:

https://modelviewculture.com/pieces/i-c ... te-my-name

It's short and to the point, give it a read. IMO it's a very good example of why diversity does matter in the tech sector.

Edit: also, have to mention

Even though our language is exempt from this effort, Han unification is particularly troubling for Bengali speakers to hear about. The rhetoric is a blast from our own colonial past, when the British referred to Indian languages pejoratively as “dialects”. Depriving their colonial subjects of distinct linguistic identities was a key tactic in justifying their brutal rule over an “uncivilized” people.

This sounds familiar. Maybe I shouldn't be so quick to assign "dialect" status to "Black vernacular" (as Wikipedia calls it, hopefully not insultingly).

[SciFiFisher]

*disclaimer* I am not a linguist. However, I believe that "black vernacular" actually can be correctly called a dialect. My rationale is thus: Black vernacular is actually American English which as been modified by African people and African Americans into a type of slang or pidgin. Wiki actually says it may be a derivative of the creole or African languages combined with American English. As such it did not exist as a separate language prior to contact with the American or English speaking people. http://en.wikipedia.org/wiki/African_Am ... ar_English

[grapes]

XOF is just trying to stay ahead of the curve. Note, these are *languages* not dialects.

XOF TRANSLATE


XOF engineers have devised eight add-ons for the popular Google Translate that will allow our readers to translate languages not supported by the popular online app. Using XOFulate, our readers will be able to decode Legalese, Victimation, Tweencant, JournalJargon, Babybabble, ItsDaBooze, Kitehigh, and Pillowtalk.

[Sigma_Orionis]

Actually GJ, while I can understand the writer's frustration with the Unicode Consortium. (I mean they took the time and effort to create encodings for Emoticons but can't be bothered to create full encodings for the different languages in Asia). We have been out of that mess for a while.

As you most probably know, we have one "special" characters in Spanish, the Ñ and its lowercase equivalent: ñ and of course: accented characters: á é í ó ú

Back in the late 19th Century/early 20th century when typewriters appeared, SUPPOSEDLY we were discouraged to use the uppercase versions of those accented characters because typewriter manufacturers could not (or would not) solve the problem of modifying typewriter keyboards properly.

After we entered the information age there were several ways to represent Spanish characters (again excluding the accented uppercase vowels), since 7 bit ASCII did not include them, it was called "National Replacement Characters" and it covered most European variations it consisted of replacing several non common characters (like '|' or '\') for the "foreign" characters, a lot of fun in Unix based systems as you can imagine .

After the rise of IBM PC compatible microcomputers, IBM's extended ASCII set (which used 8 bits to encode) included spanish characters (once again excluding the accented uppercase vowels, except for the 'É' for some reason) and both ISO 8859-1 and UTF-8 do so as well.

According to the Spanish Royal Academy we are STILL required to use accented uppercase vowels where appropriate. But, we have not used them for so long that nobody notices anymore. Few Spanish speaking people know about it and fewer even care. That's why I said that understand the writer's frustration. We have been out of our extremely mild version of that mess for a long time.

[vendic]

Read the article, the author uses all manner of reasons to vilify an attempt to make a standard that covers other languages, but isn't perfectly correct.
He argues through ignorance through most of it.
Here's a more realistic timeline of events.

ASCII and IBM's EBCDIC used 7 and 8 bit's initially.
Let's face facts, the computer WAS developed by English speaking countries, for their ease of use at a time where bit width and memory were highly expensive things to exceed without reason.
When the IBM PC came out, it used ASCII as the sole character set.
The IBM PC was a piece of crap, but, because it was IBM, a lot of people in business bought one. Proving that product excellence has no relationship with product popularity. This can still be seen today.

Unicode came out officially in 1991. Bit widths were still an issue because memory was expensive. To make things easier to port/migrate, one format of unicode was backward compatible to ASCII, the then dominant character set. It was not as the author suggests, to keep English as the dominant computer language. If they didn't include ASCII in the character set and have it compatible, he's be writing his name in English as would the rest of the world. Talk about biting the hand that feeds you.

Computers finally got cheap memory, both in RAM/ROM and long term/external storage. Unicode was then possible to to be readily implemented on computers using the extended characters. By that stage many manufactures/software houses already were conforming to the unicode set UTF-8.
Just so you're aware of it, there were also UTF-16 and UTF-32, but because they had issues with both compatibility with ASCII and cpu memory endiness were not as popular.

Like any standards organisation, it gets it's funds from members and royalties/licencing.
If you join, you get to have more of a say in what happens and the direction it goes in.
The author puts the blame for the lack of "completely accurate" characters in ASIAN countries as evidence of racism.
I call it evidence of ignorance.
The author is ignorant of the history, the reasons why we have what we have today and how we will get further.
Instead of putting the blame on his countries professional organisations who clearly did nothing to join and put their input in, or his government for their lack of interest in joining an organisation and letting them know that their language is not complete or accurate, he prefers to bitch about it in public, attribute to racism what is a pretty simple issue to understand, and, of course get paid to do so.

There's a word to describe this behavior, wanker.
It seems to fit. I should contact the consortium and have that put in as an emoticon and we can call it; Aditya.

[vendic]

Read the article, the author uses all manner of reasons to vilify an attempt to make a standard that covers other languages, but isn't perfectly correct.
He argues through ignorance through most of it.
Here's a more realistic timeline of events.

ASCII and IBM's EBCDIC used 7 and 8 bit's initially.
Let's face facts, the computer WAS developed by English speaking countries, for their ease of use at a time where bit width and memory were highly expensive things to exceed without reason.
When the IBM PC came out, it used ASCII as the sole character set.
The IBM PC was a piece of crap, but, because it was IBM, a lot of people in business bought one. Proving that product excellence has no relationship with product popularity. This can still be seen today.

Unicode came out officially in 1991. Bit widths were still an issue because memory was expensive. To make things easier to port/migrate, one format of unicode was backward compatible to ASCII, the then dominant character set. It was not as the author suggests, to keep English as the dominant computer language. If they didn't include ASCII in the character set and have it compatible, he'd be writing his name in English as would the rest of the world. Talk about biting the hand that feeds you.

Computers finally got cheap memory, both in RAM/ROM and long term/external storage. Unicode was then possible to to be readily implemented on computers using the extended characters. By that stage many manufactures/software houses already were conforming to the unicode set UTF-8. Naturally, all the unicode character sets needed to be compatible with each other so the wider variants kept their 8 bit ASCII set.
Just so you're aware of it, there were also UTF-16 and UTF-32, but because they had issues with both compatibility with ASCII and cpu memory endiness were not as popular.

Like any standards organisation, it gets it's funds from members and royalties/licencing.
If you join, you get to have more of a say in what happens and the direction it goes in.
A person drinking vodka in the street isn't going to have as much say in the direction of unicode as, say a member would. For obvious reasons.
The author puts the blame for the lack of "completely accurate" characters in ASIAN countries as evidence of racism.
I call it evidence of ignorance or stupidity.
The author is ignorant of the history, the reasons why we have what we have today and how we will get further.
Instead of putting the blame on his countries professional organisations who clearly did nothing to join and put their input in, or his government for their lack of interest in joining an organisation and letting them know that their language is not complete or accurate, he prefers to bitch about it in public, attribute to racism to what is a pretty simple thing to understand, and, of course he gets paid to do so as well as publicity.

There's a word to describe this behavior, wanker.
It seems to fit. I should contact the consortium and have that put in as an emoticon and we can call it; Aditya.

[Cyborg Girl]

a) The author, in case you missed it, is a software engineer. You can bet he knows at least that much about Unicode's history, seeing as what you outlined is what anyone will get who has read any recent book on Python or such.

b) Can we please get over the "OMG racism" thing? This is about tacit bias, not deliberate bigotry. Call it what you want, it is out there.

c) A 32-bit int is an awful lot of possible characters, even given the number of written languages on Earth. I don't know about you; but it seems bizarre to me to be trying to conserve space by buggering up common extant languages, regardless of which languages, while adding customizable smiley faces.

d) What the fuck does vodka have to do with it?

e)

Like any standards organisation, it gets it's funds from members and royalties/licencing.
If you join, you get to have more of a say in what happens and the direction it goes in.

... And given the amount of power that standards organizations have, how is this not a problem? The parties that have a say will, for the most part, be the ones that benefit most. If you have to pay to join, then the parties with the most money will benefit the most. When you're talking about standards that are supposed to be universal, and that can impact things like employment opportunities, don't you think that's not necessarily a good thing?

Yes, I know this is the normal state of affairs. That doesn't mean it's a desirable state of affairs.

[vendic]

a) The author, in case you missed it, is a software engineer. You can bet he knows at least that much about Unicode's history, seeing as what you outlined is what anyone will get who has read any recent book on Python or such.

https://modelviewculture.com/authors/aditya-mukerjee

He is young. Not old enough to have been there when this happened, as such, he is the one that would have to have read up on it.
I lived through it. Seeing as we're throwing qualifications around, I'm an engineer as well. I worked in hardware and software years before unicode was brought in and watched it develop.

b) Can we please get over the "OMG racism" thing? This is about tacit bias, not deliberate bigotry. Call it what you want, it is out there.

So is the propensity to make something about race, inherent racism, sexism, inherent sexism etc about things that had nothing to do with it.

c) A 32-bit int is an awful lot of possible characters, even given the number of written languages on Earth. I don't know about you; but it seems bizarre to me to be trying to conserve space by buggering up common extant languages, regardless of which languages, while adding customizable smiley faces.

It is a lot. Never wrote anything different.
You however didn't read what I wrote or failed to understand it.
It did not start out as 32bit, it started out as 16bit, then 8bit expandable ASCII compatible.
That means the author is simply ignorant when he complains about it having the ASCII character set inbuilt.
Let's say the standard was 100bits wide.
Does that mean that on it's release it will have everything that everyone ever wants without fault.
No. It means it's expandable. To fill those characters, a case needs to be made for their inclusion. Bit width doesn't automatically mean it's there, it means it can be there.

d) What the fuck does vodka have to do with it?

Same thing that racism, direct, implied or as in this case, imaginary has to do with unicode not covering every character used in Earth's languages.

e)

Like any standards organisation, it gets it's funds from members and royalties/licencing.
If you join, you get to have more of a say in what happens and the direction it goes in.

... And given the amount of power that standards organizations have, how is this not a problem? The parties that have a say will, for the most part, be the ones that benefit most. If you have to pay to join, then the parties with the most money will benefit the most. When you're talking about standards that are supposed to be universal, and that can impact things like employment opportunities, don't you think that's not necessarily a good thing?

Yes, I know this is the normal state of affairs. That doesn't mean it's a desirable state of affairs.

How is what a problem?
That they openly allow people to join their organisation, take submissions from members as well as the public and tried to implement a character set that includes as much as they can in it.
Did the author ever contact them and point out that there are characters missing?
Since he never stated it, I'm going to guess no. If he did, he'd be complaining that he'd addressed this issue and they hadn't done anything about it. That would have been a legitimate complaint.

How the hell is a character set missing certain characters going to affect job opportunities?
Seriously.
If the person is in a country that has the full set for them, it makes no difference.
If they are in a country where they haven't the full character set, then none of them have it.

Since you seem to be ignorant of the world of standards, let me explain a few things. They don't work unless you can get a large section of the people that use the scope of the standard to agree to it and to do that you will need both funding and business influence. People in the industry are meant to contribute to further the standard. if they don't address something, the general assumption, for good reason is: don't touch it since it ain't broke.
People in the industry for a character set in this case would be a countries government, it it affects them, a professional organisation, if it affects them. Hell, even individuals can make comments if they want to, if it affects them.

A character set is only useful to people who want to use it.

Bottom line is this, unless someone has tried to get them to amend it, has written to them regularly to point out the problems, and has not got any responses as a paid member, the argument that they have not implemented something due to reason X,Y or Z, is disingenuous. You can't expect one standards body to know and understand everything about other groups and be blamed for ignorance (or in this case, bias), if those very groups themselves have not taken any appreciable steps to address their concerns in the first place.
Have they?
If they have, then lets see the paper trail before I condemn a standards body.
If they haven't, then the complaining groups are the problem, not the standards body.

It seems people now just expect things to be done for them, rather than taking basic initiatives to do something, even if it's as simple as writing to the body itself to get their issues addressed. Instead, they write to others.

In the restaurant industry it's common to say, "If you like the food and service, tell someone else. If you don't, tell us".
Same principle applies here. It's just plain rude not to do so.

Finally, as someone that has written standards, has made suggestions to other bodies regarding issues with standards, I can tell the author has little understanding of the process or appreciation of the complexities involved. Hell, right now I am in the process of writing one. I take as much input as I can get. If you don't input, you don't get to complain I didn't consider your particular case.

[Sigma_Orionis]

Read the article, the author uses all manner of reasons to vilify an attempt to make a standard that covers other languages, but isn't perfectly correct.
He argues through ignorance through most of it.

While the article reeks of racial resentment, and he does argue out of ignorance. I think he's mainly pissed off because he considers that the Unicode Consortium spent more efforts in encoding emoticons than multiple byte Asian character sets

Here's a more realistic timeline of events.

ASCII and IBM's EBCDIC used 7 and 8 bit's initially.
Let's face facts, the computer WAS developed by English speaking countries, for their ease of use at a time where bit width and memory were highly expensive things to exceed without reason.

True. I don't think it it was even practical. I still remember in the 80s a 500 key chinese keyboard being featured in a hobbyist magazine of the day. The Japanese didn't have those problems because had a character set that could be practically encoded in 8 bits or less Katakana

When the IBM PC came out, it used ASCII as the sole character set.

Nitpick, it used a proprietary 8-Bit extension of ASCII. Standarized ASCII was and still is a 7 bit character set just like everyone else in the Micro-Computer business in those days.

The IBM PC was a piece of crap, but, because it was IBM, a lot of people in business bought one. Proving that product excellence has no relationship with product popularity. This can still be seen today.

Irrelevant. While it's true that the IBM 5150 was a "me too" product created by IBM in a hurry to enter the Micro-Computer market it was not markedly worse than any of the Microcomputers of the day. I owned a Tandy/Radio-Shack TRS-80 (aka "Trash-80") whose design and quality were really sub-par. I also remember using a CompuColor II and had the Keyboard/CPU unit literally blow up in front of me while playing Star Trek I do have to admit that MSDOS 1.0 was pathetic. But, let's remember how well Apple did with the Apple III......

Unicode came out officially in 1991. Bit widths were still an issue because memory was expensive. To make things easier to port/migrate, one format of unicode was backward compatible to ASCII, the then dominant character set. It was not as the author suggests, to keep English as the dominant computer language. If they didn't include ASCII in the character set and have it compatible, he's be writing his name in English as would the rest of the world. Talk about biting the hand that feeds you.

Quite true, Multiple Byte character sets were and to some point still are (just try to convert a single byte character database to multiple byte, it's a lot of work) a royal pain in the ass. I don't recall the author suggesting of a "deliberate attempt" to keep English as the dominant computer language (although it was and still is. And NO, I DON'T HAVE AN ISSUE WITH THAT.) Although he makes it quite clear that he thinks that organizations like Unicode don't pay enough attention to his beef.

Computers finally got cheap memory, both in RAM/ROM and long term/external storage. Unicode was then possible to to be readily implemented on computers using the extended characters. By that stage many manufactures/software houses already were conforming to the unicode set UTF-8.
Just so you're aware of it, there were also UTF-16 and UTF-32, but because they had issues with both compatibility with ASCII and cpu memory endiness were not as popular.

AFAIK, there's a fair amount of support for UTF-32. I tend to install Oracle Databases with UTF-32 by default, if only because converting them from ISO-8859 or lower is a real PITA (it usually requires reinstalling the RDBMS)

Like any standards organisation, it gets it's funds from members and royalties/licencing.
If you join, you get to have more of a say in what happens and the direction it goes in.
The author puts the blame for the lack of "completely accurate" characters in ASIAN countries as evidence of racism.
I call it evidence of ignorance.
The author is ignorant of the history, the reasons why we have what we have today and how we will get further.
Instead of putting the blame on his countries professional organisations who clearly did nothing to join and put their input in, or his government for their lack of interest in joining an organisation and letting them know that their language is not complete or accurate, he prefers to bitch about it in public, attribute to racism what is a pretty simple issue to understand, and, of course get paid to do so.

There's a word to describe this behavior, wanker.
It seems to fit. I should contact the consortium and have that put in as an emoticon and we can call it; Aditya.

That's a good point, and I tend to agree, see this, while it's geared towards the Japanese standard, it asserts that while Unicode's "Han Unification" is reputed to be a "Western attempt" to ram a Western Oriented standard down Asian throats, it has considerable input from Asian interests. It would help if you didn't react so much to the author's racial resentment though.

[vendic]

That's a good point, and I tend to agree, see this, while it's geared towards the Japanese standard, it asserts that while Unicode's "Han Unification" is reputed to be a "Western attempt" to ram a Western Oriented standard down Asian throats, it has considerable input from Asian interests. It would help if you didn't react so much to the author's racial resentment though.

But, it's fun

[Sigma_Orionis]

You must be a friend of Zee's

[Cyborg Girl]

@vendic - we have 32 bits of possible characters now. And it's being used for emoticons instead of better language support. That is the key issue I got out of the article.

But, screw it, I'm done arguing here. Feel free to do a victory dance if you want.

[vendic]

@vendic - we have 32 bits of possible characters now. And it's being used for emoticons instead of better language support. That is the key issue I got out of the article.

But, screw it, I'm done arguing here. Feel free to do a victory dance if you want.

You're missing the point. If you aren't told that there are language support issues, then you aren't making a decision to sacrifice one over the other. The point has also been made that there are plenty of spare codes to do all if it.
The question then is, why isn't it being used?
There are two likely possibilities:
1) They have no clue that they haven't filled all the language requirements.
2) They chose to ignore language in favor of emoticons and Klingon.

(1) Is where I'm arguing the problem is.
The author is arguing (2) even though he and you acknowledge that there are plenty of codes spare to do everything so there is no reason to sacrifice one over another.

Feel free to make your case that they are willfully dismissing other languages, but, I'll ask you to supply proof that they are. That's fair, isn't it?
If you can't come up with any, then the reality more likely is that, the author is wrong, and all that needs be done is point out to the standards group the language issues. Preferably using channels they have to do this rather than chastising them in public as is the case now.

I don't do *victory dance* winning an argument.
I do *happy dance* when I realise I may have taught someone something new and get them to understand that there may be more to an issue than assuming that some group is corrupt/racist/biased knowingly or not.

[vendic]

Here: http://www.unicode.org/faq/prop_new_characters.html

" Where do I find information on how to use proposed characters for future versions of the standard?

A: Those characters are not yet standardized, and have not yet finished their rounds of international ISO balloting. Implementers must be very cautious about trying to implement tentatively accepted characters before their final publication. However, information about the proposed characters is generally available in the working proposal documents. Anyone can browse the L2 document list which contains new character proposals. Public proposal documents can also be found at the JTC1/SC2/WG2 site, or at Michael Everson's standards site. Links to many proposal documents can also be found via the Unicode roadmaps."

The above gives links to places where the PUBLIC can put forth proposals.
They are pretty transparent giving details of the proposals, you can read them.
Feel free to see if they have any for the authors issue and if they have a proposed roadmap to address it.

I posted previously, standards groups take suggestions from industry and anyone that may be impacted. If no one raised an issue, how the hell are they meant to do anything about it?

Now, if someone has an issue they can report it, if no one comes up with a proposal to correct that issue, how is it going to get implemented?
Someone suggested and then someone else proposed Klingon and emoticon's, they then fitted it in. By someone, I mean lots of someone's.
Where's the proposals for the author's issue?

See where I'm going with this. Ignorance of the process is not an excuse. You agreeing with someone that is ignorant of the process is just the blind leading the blind. Go and check for yourself. Don't take my word for it.


http://www.unicode.org/pending/proposals.html

There's all the information above that you need to submit your own proposal. If you have one, go for it.
If you don't, but want to complain that a character isn't in there, then it's not the committee's fault.

[vendic]

Last but not least, I figured I'd take the time to comment on the authors page and point out that unicode has a proposal page, road map and allow the public to make proposals, which is completely contrary to what the article implied.
I wanted to put the links to the standards website that allows the author to make a proposal to change the state of things as they are.

Guess what?
They don't provide an open method for the public to make comments so they can improve their knowledge and get told if they are spreading false information in order to correct it.
Funny, the unicode standards group does.
Hypocrites.

*Edited to add*
I checked the proposed scripts and found non for Bengali.
I checked the rejected scripts for unicode, and found none for Bengali.

IOW, NO ONE has proposed any amendments and non were rejected.

Feel free to try and contact the author, seeing as he is an expert in the field of unicode and let him know of his ignorance.
I tried to comment, couldn't.
Not my problem if he isn't even interested enough to do anything to change the situation and it affects him so much.