WebStack

docs/CHARSET.txt

732:7f1f02b485f8
2007-11-12 paulb [project @ 2007-11-12 00:50:03 by paulb] Introduced base classes for common authentication activities. Made cookie usage "safe" for usernames containing ":" characters. Added support for OpenID signatures.
     1 Unicode and Character Sets in WebStack
     2 --------------------------------------
     3 
     4 Unicode text should be converted to the chosen character set (encoding) when
     5 written to the response stream.
     6 
     7 Classic Python strings are written directly to the response stream without
     8 encoding.
     9 
    10 Character Set Semantics in WebStack
    11 -----------------------------------
    12 
    13 Character sets (or encodings) are relevant in two areas:
    14 
    15  * The encoding of output data.
    16  * The processing of input data.
    17 
    18 When producing HTML pages containing form fields and interpreting the values of
    19 such fields from a request body, it is necessary to know...
    20 
    21  * The character set used to encode the values sent by the browser. This is
    22    typically determined by...
    23 
    24  * The character set used to encode the HTML page from which the field values
    25    originated.
    26 
    27 It is therefore also necessary to remain consistent in the usage of character
    28 sets when specifying content types. WebStack enforces the following rules:
    29 
    30  * Where the request content type specifies a character set, this is used to
    31    decode the request body parameters unless explicitly overridden.
    32 
    33  * Where the request content type does not specify a character set, a default
    34    character set is used to decode the request body parameters unless
    35    overridden.
    36 
    37  * No conversion is done at the request stream level, since information about
    38    the character set may be missing and the application may wish to override
    39    any default explicitly at a higher level (such as when it gets request body
    40    parameters).
    41 
    42  * Where the response content type specifies a character set, this is used to
    43    encode Unicode response data (eg. HTML pages).
    44 
    45  * Where the response content type does not specify a character set, a default
    46    character set is used to encode Unicode response data (eg. HTML pages).
    47 
    48 Restrictions in and Omissions from Standards
    49 --------------------------------------------
    50 
    51 The encoding of character sets such as UTF-16 in HTTP POST request body
    52 messages of content/media type application/x-www-form-urlencoded is not
    53 properly standardised. Therefore, it is highly recommended that UTF-8 be used
    54 as an encoding should the various single byte encodings (eg. ISO-8859-1) not
    55 cover the range of characters to be displayed and received.
    56 
    57 Framework Behaviour
    58 -------------------
    59 
    60 The Java Servlet API imposes restrictions on decoding request body parameters
    61 by stating that the character encoding (ServletRequest.setCharacterEncoding)
    62 must be set before any reading of the request body is attempted.