分类： C/C++
2011-03-25 16:04:41
Using Expat
by Clark Cooper
September 01, 1999
What is expat?
Expat is a library, written in C, for parsing XML documents. It's the underlying XML parser for the open source Mozilla project, perl's XML::Parser, and other open-source XML parsers. As demonstrated in mybenchmark article, it's very fast. It also sets a high standard for reliability, robustness and correctness.
This library is the creation of James Clark, who's also given us groff (an nroff look-alike), Jade (an implemention of ISO's DSSSL stylesheet language for SGML), XP (a Java XML parser package), and XT (a Java XSL engine). James was also the technical lead on the XML Working Group at W3 that produced the XML specification. Many of these packages are available from ftp://ftp.jclark.com/pub, including expat. There's also a test version , which has newer features, but which may not be as robust as the non-test version. This article is based on a test version, Version 19990709.
Overview of Expat
Expat is a stream-oriented parser. You register callback (or handler) functions with the parser and then start feeding it the document. As the parser recognizes parts of the document, it will call the appropriate handler for that part (if you've registered one). The document is fed to the parser in pieces, so you can start parsing before you have the whole document. This also allows you to parse really huge documents that won't fit into memory.
Expat can be intimidating due to the many kinds of handlers and options you can set. But you only need to learn four functions in order to do 80% of what you'll want to do with it:
XML_ParserCreate
Create a new parser object.
XML_SetElementHandler
Set handlers for start and end tags.
XML_SetCharacterDataHandler
Set handler for text.
XML_Parse
Pass a buffer full of document to the parser
These functions and others are described in the reference part of this article. The reference section also describes in detail the parameters passed to the different types of handlers.
This Zip file contains the Makefile and source code for examples used in this article.
Let's look at a very simple example program that only uses three of the above functions. (It doesn't need to set a character handler.) The program outline.c prints an element outline, indenting child elements to distinguish them from the parent element that contains them. The start handler does all the work. It prints two indenting spaces for every level of ancestor elements, then it prints the element and attribute information. Finally it increments the global Depth variable.
int Depth;
void
start(void *data, const char *el, const char **attr) {
int i;
for (i = 0; i < Depth; i++)
printf(" ");
printf("%s", el);
for (i = 0; attr[i]; i += 2) {
printf(" %s='%s'", attr[i], attr[i + 1]);
}
printf("\n");
Depth++;
} /* End of start handler */
The end tag simply does the bookkeeping work of decrementing the Depth.
void
end(void *data, const char *el) {
Depth--;
} /* End of end handler */
After creating the parser, the main program just has the job of shoveling the document to the parser so that it can do its work.
Building expat
One of the problems with using expat is that isn't packaged as a library. Instead there are four separate object files that you have to link into your application. The Makefile that builds the sample applications in this article can be used as a template.
Compile time conditionals
There are a few compiletime macros that control how the compiled expat behaves:
XML_UNICODE
Use UTF-16 internally and pass strings to application using UTF-16 instead of UTF-8. This changes the type definition of XML_Char, which would otherwise be defined as char.
XML_UNICODE_WCHAR_T
Use UTF-16 internally as declared as wchar_t from
XML_DTD
Include code to parse external DTD.
XML_NS
Do lexical checking of namespaces
XML_BYTE_ORDER
Set this to "12" for little-endian machines (machines that have the least significant byte first) and to "21" for big-endian (most significant byte first.)
XML_MIN_SIZE
Makes a parser that's smaller but that, in general, will run slower.
If your system doesn't have memmove, but does have bcopy, then you'll want to have a macro that redfines memmove to bcopy. There's a Makefile macro that does this in the sample Makefile, XP_MM. You'll have to uncomment its definition in order to have it take effect.
Working with Expat
As I mentioned in the overview section, the document is fed to the parser a piece at a time. It is completely up to the calling application how much of the document to fit into a piece. The sample program, line demonstrates this. It passes a line at a time to the parser and then reports start, end, text, and processing instruction events. By interactively typing in a document into this program, you may start to obtain an intuitive feel for how the parser is working.
Walking through a document hierarchy with a stream oriented parser will require a good stack mechanism in order to keep track of current context. For instance, to answer the simple question, "What element does this text belong to?" requires a stack, since the parser may have descended into other elements that are children of the current one and has encountered this text on the way out.
The things you're likely to want to keep on a stack are the currently opened element and its attributes. You push this information onto the stack in the start handler and you pop it off in the end handler.
For some tasks, it is sufficient to just keep information on what the depth of the stack is (or would be if you had one.) The outline program shown above presents one example. Another such task would be skipping over a complete element. When you see the start tag for the element you want to skip, you set a skip flag and record the depth at which the element started. When the end tag handler encounters the same depth, the skipped element has ended and the flag may be cleared. If you follow the convention that the root element starts at 1, then you can use the same variable for skip flag and skip depth.
void
init_info(Parseinfo *info) {
info->skip = 0;
info->depth = 1;
/* Other initializations here */
} /* End of init_info */
void
rawstart(void *data, const char *el, const char **attr) {
Parseinfo *inf = (Parseinfo *) data;
if (! inf->skip) {
if (should_skip(inf, el, attr)) {
inf->skip = inf->depth;
}
else
start(inf, el, attr); /* This does rest of start handling */
}
inf->depth++;
} /* End of rawstart */
void
rawend(void *data, const char *el) {
Parseinfo *inf = (Parseinfo *) data;
inf->depth--;
if (! inf->skip)
end(inf, el); /* This does rest of end handling */
if (inf->skip == inf->depth)
inf->skip = 0;
} /* End rawend */
Notice in the above example the difference in how depth is manipulated in the start and end handlers. The end tag handler should be the mirror image of the start tag handler. This is necessary to properly model containment. Since, in the start tag handler, we incremented depth after the main body of start tag code, then in the end handler, we need to manipulate it before the main body. If we'd decided to increment it first thing in the start handler, then we'd have had to decrement it last thing in the end handler.
Communicating between handlers
In order to be able to pass information between different handlers without using globals, you'll need to define a data structure to hold the shared variables. You can then tell expat (with the XML_SetUserData function) to pass a pointer to this structure to the handlers. This is typically the first argument received by most handlers.
Namespace Processing
When the parser is created using the XML_ParserCreateNS, function, expat performs namespace processing. Under namespace processing, expat consumes xmlns and xmlns:... attributes, which declare namespaces for the scope of the element in which they occur. This means that your start handler will not see these attributes. Your application can still be informed of these declarations by setting namespace declaration handlers with .
Element type and attribute names that belong to a given namespace are passed to the appropriate handler in expanded form. This expanded form is a concatenation of the namespace URI, the separator character (which is the 2nd argument to XML_ParserCreateNS), and the local name (i.e. the part after the colon). Names with undeclared prefixes are passed through to the handlers unchanged, with the prefix and colon still attached. Unprefixed attribute names are never expanded, and unprefixed element names are only expanded when they are in the scope of a default namespace.
You can set handlers for the start of a namespace declaration and for the end of a scope of a declaration with the XML_SetNamespaceDeclHandler function. The StartNamespaceDeclHandler is called prior to the start tag handler and the EndNamespaceDeclHandler is called before the corresponding end tag that ends the namespace's scope. The namespace start handler gets passed the prefix and URI for the namespace. For a default namespace declaration (xmlns='...'), the prefix will be null. The URI will be null for the case where the default namespace is being unset. The namespace end handler just gets the prefix for the closing scope.
These handlers are called for each declaration. So if, for instance, a start tag had three namespace declarations, then the StartNamespaceDeclHandler would be called three times before the start tag handler is called, once for each declaration.
The namespace.c example demonstrates the use of these features. Like outline.c, it produces an outline, but in addition it annotates when a namespace scope starts and when it ends. This example also demonstrates use of application user data.
Character Encodings
While XML is based on Unicode, and every XML processor is required to recognized UTF-8 and UTF-16 (1 and 2 byte encodings of Unicode), other encodings may be declared in XML documents or entities. For the main document, an XML declaration may contain an encoding declaration:
External parsed entities may begin with a text declaration, which looks like an XML declaration with just an encoding declaration:
With expat, you may also specify an encoding at the time of creating a parser. This is useful when the encoding information may come from a source outside the document itself (like a higher level protocol.)
There are four built-in encodings in expat:
Anything else discovered in an encoding declaration or in the protocol encoding specified in the parser constructor, triggers a call to the UnknownEncodingHandler. This handler gets passed the encoding name and a pointer to an XML_Encoding data structure. Your handler must fill in this structure and return 1 if it knows how to deal with the encoding. Otherwise the handler should return 0. The handler also gets passed a pointer to an optional application data structure that you may indicate when you set the handler.
Expat places restrictions on character encodings that it can support by filling in the XML_Encodingstructure. include file:
1. Every ASCII character that can appear in a well-formed XML document must be represented by a single byte, and that byte must correspond to it's ASCII encoding (except for the characters $@\^'{}~)
2. Characters must be encoded in 4 bytes or less.
3. All characters encoded must have Unicode scalar values less than or equal to 65535 (0xFFFF)This does not apply to the built-in support for UTF-16 and UTF-8
4. No character may be encoded by more that one distinct sequence of bytes
XML_Encoding contains an array of integers that correspond to the 1st byte of an encoding sequence. If the value in the array for a byte is zero or positive, then the byte is a single byte encoding that encodes the Unicode scalar value contained in the array. A -1 in this array indicates a malformed byte. If the value is -2, -3, or -4, then the byte is the beginning of a 2, 3, or 4 byte sequence respectively. Multi-byte sequences are sent to the convert function pointed at in theXML_Encoding structure. This function should return the Unicode scalar value for the sequence or -1 if the sequence is malformed.
One pitfall that novice expat users are likely to fall into is that although expat may accept input in various encodings, the strings that it passes to the handlers are always encoded in UTF-8. Your application is responsible for any translation of these strings into other encodings.
Handling External Entity References
Expat does not read or parse external entities directly. Note that any external DTD is a special case of an external entity. If you've set no ExternalEntityRefHandler, then external entity references are silently ignored. Otherwise, it calls your handler with the information needed to read and parse the external entity.
Your handler isn't actually responsible for parsing the entity, but it is responsible for creating a subsidiary parser with XML_ExternalEntityParserCreate that will do the job. This returns an instance of XML_Parser that has handlers and other data structures initialized from the parent parser. You may then use XML_Parse or XML_ParseBuffer calls against this parser. Since external entities my refer to other external entities, your handler should be prepared to be called recursively.
Parsing DTDs
In order to parse parameter entities, the macro XML_DTD, must be defined when expat is compiled. In addition, after creating the parser and before starting the parse, you must callXML_SetParamEntityParsing with one of the following arguments:
XML_PARAM_ENTITY_PARSING_NEVER
Don't parse parameter entities or the external subset
XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE
Parse parameter entites and the external subset unless standalone was set to "yes" in the XML declaration.
XML_PARAM_ENTITY_PARSING_ALWAYS
Always parse parameter entities and the external subset
In order to read an external subset, you also have to set an external entity reference handler as described above.
Expat Function Reference
|
Expat Functions |
|||
|
|
|
|
XML_ErrorString |
Parser Creation
XML_Parser
XML_ParserCreate(const XML_Char*encoding)
Construct a new parser. If encoding is non-null, it specifies a character
encoding to use for the document. This overrides the document encoding
declaration. There are four built-in encodings:
Any other value will invoke a call to the UnknownEncodingHandler.
XML_Parser
XML_ParserCreateNS(const XML_Char*encoding, XML_Char sep)
Constructs a new parser that has namespace processing in effect. Namespace
expanded element names and attribute names are returned as a concatenation of
the namespace URI, sep, and the local part of the name. This means
that you should pick a character for sep that can't be part of
a legal URI.
XML_Parser
XML_ExternalEntityParserCreate(XML_Parser p, const XML_Char *context, const
XML_Char *encoding)
Construct a new XML_Parser object for parsing an external general entity.
Context is the context argument passed in a call to a ExternalEntityRefHandler.
Other state information such as handlers, user data, namespace processing is
inherited from the parser passed as the 1st argument. So you shouldn't need to
call any of the behavior changing functions on this parser (unless you want it
to act differently than the parent parser.)
void
XML_ParserFree(XML_Parser p)
Free memory used by the parser. Your application is responsible for
freeing any memory associated with UserData.
Parsing
int
XML_Parse(XML_Parser p, const char *s, int len, int isFinal)
Parse some more of the document. The string s is a buffer
containing part (or perhaps all) of the document. The number of bytes of s that
are part of the document is indicated by len. This means thats doesn't have
to be null terminated. It also means that if len is larger
than the number of bytes in the block of memory that s points at,
then a memory fault is likely. The isFinal parameter
informs the parser that this is the last piece of the document. Frequently, the
last piece is empty (i.e. len is zero.) If
a parse error occurred, it returns 0. Otherwise it returns a non-zero value.
int
XML_ParseBuffer(XML_Parser p, int len, int isFinal)
This is just like XML_Parse, except in this case expat provides the
buffer. By obtaining the buffer from expat with the XML_GetBuffer function, the
application can avoid double copying of the input.
void
*XML_GetBuffer(XML_Parser p, int len)
Obtain a buffer of size len to read a
piece of the document into. A NULL value is returned if expat can't allocate
enough memory for this buffer. This has to be called prior to every call
to XML_ParseBuffer. A typical use would look like this:
for (;;) {
int bytes_read;
void *buff = XML_GetBuffer(p, BUFF_SIZE);
if (buff == NULL) {
/* handle error */
}
bytes_read = read(docfd, buff, BUFF_SIZE);
if (bytes_read < 0) {
/* handle error */
}
if (! XML_ParseBuffer(p, bytes_read, bytes_read == 0)) {
/* handle parse error */
}
if (bytes_read == 0)
break;
}
Handler Setting
Although handlers are typically set prior to parsing and left alone, an application may choose to set or change the handler for a parsing event while the parse is in progress. For instance, your application may choose to ignore all text not descended from a para element. One way it could do this is to set the character handler when a para start tag is seen, and unset it for the corresponding end tag.
A handler may be unset by providing a NULL pointer to the appropriate handler setter. None of the handler setting functions have a return value.
Your handlers will be receiving strings in arrays of type XML_Char. This type is defined in xmlparse.h and is conditional upon the setting of either of the XML_UNICODE macros. If neither of these is set, then XML_Char contains characters encoding UTF-8. Otherwise you'll be receiving UTF-16 in the form of either unsigned short or wchar_t characters.
Note that you'll receive them in this form independent of the original encoding of the document. Elsewhere in this document, I may make this point by simply referring to UTF-8.
XML_SetElementHandler(XML_Parser p,
XML_StartElementHandler start,
XML_EndElementHandler end);
typedef void
(*XML_StartElementHandler)(void *userData,
const XML_Char *name,
const XML_Char **atts);
typedef void
(*XML_EndElementHandler)(void *userData,
const XML_Char *name);
Set handlers for start and end tags. Attributes are passed to the start handler as a pointer to a vector of char pointers. Each attribute seen in a start (or empty) tag occupies 2 consecutive places in this vector: the attribute name followed by the attribute value. These pairs are terminated by a null pointer.
XML_SetCharacterDataHandler(XML_Parser p,
XML_CharacterDataHandler charhndl)
typedef void
(*XML_CharacterDataHandler)(void *userData,
const XML_Char *s,
int len);
Set a text handler. The string your handler receives is NOT zero terminated. You have to use the length argument to deal with the end of the string. A single block of contiguous text free of markup may still result in a sequence of calls to this handler. In other words, if you're searching for a pattern in the text, it may be split across calls to this handler.
XML_SetProcessingInstructionHandler(XML_Parser p,
XML_ProcessingInstructionHandler proc)
typedef void
(*XML_ProcessingInstructionHandler)(void *userData,
const XML_Char *target,
const XML_Char *data);
Set a handler for processing instructions. The target is the first word in the processing instruction. The data is the rest of the characters in it after skipping all whitespace after the initial word.
XML_SetCommentHandler (XML_Parser p,
XML_CommentHandler cmnt)
typedef void
(*XML_CommentHandler)(void *userData,
const XML_Char *data);
Set a handler for comments. The data is all text inside the comment delimiters.
XML_SetCdataSectionHandler(XML_Parser p,
XML_StartCdataSectionHandler start,
XML_EndCdataSectionHandler end)
typedef void
(*XML_StartCdataSectionHandler)(void *userData);
typedef void
(*XML_EndCdataSectionHandler)(void *userData);
Sets handlers that get called at the beginning and end of a CDATA section.
XML_SetDefaultHandler(XML_Parser p,
XML_DefaultHandler hndl)
typedef void
(*XML_DefaultHandler)(void *userData,
const XML_Char *s,
int len);
Sets a handler for any characters in the document which wouldn't otherwise be handled. This includes both data for which no handlers can be set (like some kinds of DTD declarations) and data which could be reported but which currently has no handler set. Note that a contiguous piece of data that is destined to be reported to the default handler may actually be reported over several calls to the handler. Setting the handler with this call has the side effect of turning off expansion of references to internally defined general entities. Instead these references are passed to the default handler.
XML_SetDefaultHandlerExpand(XML_Parser p,
XML_DefaultHandler hndl)
This sets a default handler, but doesn't affect expansion of internal entity references.
XML_SetExternalEntityRefHandler(XML_Parser p,
XML_ExternalEntityRefHandler hndl)
typedef int
(*XML_ExternalEntityRefHandler)(XML_Parser parser,
const XML_Char *context,
const XML_Char *base,
const XML_Char *systemId,
const XML_Char *publicId);
Set an external entity reference handler. This handler is also called for processing an external DTD subset if parameter entity parsing is in effect. (See XML_SetParamEntityParsing )
The base parameter is the base to use for relative system identifiers. It is set by XML_SetBase and may be null. The public id parameter is the public id given in the entity declaration and may be null. The system id is the system identifier specified in the entity declaration and is never null.
There are a couple of ways in which this handler differs from others. First, this handler returns an integer. A non-zero value should be returned for successful handling of the external entity reference. Returning a zero indicates failure, and causes the calling parser to return an XML_ERROR_EXTERNAL_ENTITY_HANDLING error.
Second, instead of having userData as its first argument, it receives the parser that encountered the entity reference. This, along with the context parameter, may be used as arguments to a call toXML_ExternalEntityParserCreate. Using the returned parser, the body of the external entity can be recursively parsed.
Since this handler may be called recursively, it should not be saving information into global or static variables.
XML_SetUnknownEncodingHandler(XML_Parser p,
XML_UnknownEncodingHandler enchandler,
void *encodingHandlerData)
typedef int
(*XML_UnknownEncodingHandler)(void *encodingHandlerData,
const XML_Char *name,
XML_Encoding *info);
Set a handler to deal with encodings other than the built in set. If the handler knows how to deal with an encoding with the given name, it should fill in the info data structure and return 1. Otherwise it should return 0.
typedef struct {
int map[256];
void *data;
int (*convert)(void *data, const char *s);
void (*release)(void *data);
} XML_Encoding;
The map array contains information for every possible possible leading byte in a byte sequence. If the corresponding value is >= 0, then it's a single byte sequence and the byte encodes that Unicode value. If the value is -1, then that byte is invalid as the initial byte in a sequence. If the value is -n, where n is an integer > 1, then n is the number of bytes in the sequence and the actual conversion is accomplished by a call to the function pointed at by convert. This function may return -1 if the sequence itself is invalid. The convert pointer may be null if there are only single byte encodings. The data parameter passed to the convert function is the data pointer from XML_Encoding. The string s isNOT null terminated and points at the sequence of bytes to be converted.
The function pointed at by release is called by the parser when it is finished with the encoding. It may be null.
XML_SetNamespaceDeclHandler(XML_Parser p,
XML_StartNamespaceDeclHandler start,
XML_EndNamespaceDeclHandler end)
typedef void
(*XML_StartNamespaceDeclHandler)(void *userData,
const XML_Char *prefix,
const XML_Char *uri);
typedef void
(*XML_EndNamespaceDeclHandler)(void *userData,
const XML_Char *prefix);
Set handlers for namespace declarations. Namespace declarations occur inside start tags. But the namespace declaration start handler is called before the start tag handler for each namespace declared in that start tag. The corresponding namespace end handler is called after the end tag for the element the namespace is associated with.
XML_SetUnparsedEntityDeclHandler(XML_Parser p,
XML_UnparsedEntityDeclHandler h)
typedef void
(*XML_UnparsedEntityDeclHandler)(void *userData,
const XML_Char *entityName,
const XML_Char *base,
const XML_Char *systemId,
const XML_Char *publicId,
const XML_Char *notationName);
Set a handler that receives declarations of unparsed entities. These are entity declarations that have a notation (NDATA) field:
So for this example, the entityName would be "logo", the systemId would be "images/logo.gif" and notationName would be "gif". For this example the publicId parameter is null. The base parameter would be whatever has been set with XML_SetBase. If not set, it would be null.
XML_SetNotationDeclHandler(XML_Parser p,
XML_NotationDeclHandler h)
typedef void
(*XML_NotationDeclHandler)(void *userData,
const XML_Char *notationName,
const XML_Char *base,
const XML_Char *systemId,
const XML_Char *publicId);
Set a handler that receives notation declarations.
XML_SetNotStandaloneHandler(XML_Parser p,
XML_NotStandaloneHandler h)
typedef int
(*XML_NotStandaloneHandler)(void *userData);
Set a handler that is called if the document is not "standalone". This happens when there is an external subset or a reference to a parameter entity, but does not have standalone set to "yes" in an XML declaration. If this handler returns 0, then the parser will throw an XML_ERROR_NOT_STANDALONE error.
Parse position and error reporting functions
These are the functions you'll want to call when the parse functions return 0, although the position reporting functions are useful outside of errors. The position reported is that of the first of the sequence of characters that generated the current event (or the error that caused the parse functions to return 0.)
enum XML_Error
XML_GetErrorCode(XML_Parser p)
Return what type of error has occurred.
const XML_LChar
*XML_ErrorString(int code)
Return a string describing the error corresponding to code. The code
should be one of the enums that can be returned from XML_GetErrorCode.
long
XML_GetCurrentByteIndex(XML_Parser p)
Return the byte offset of the position.
int
XML_GetCurrentLineNumber(XML_Parser p)
Return the line number of the position.
int
XML_GetCurrentColumnNumber(XML_Parser p)
Return the offset, from the beginning of the current line, of the
position.
Miscellaneous functions
The functions in this section either obtain state information from the parser or can be used to dynamically set parser options.
XML_SetUserData(XML_Parser
p, void *userData)
This sets the user data pointer that gets passed to handlers.
void *
XML_GetUserData(XML_Parser p)
This returns the user data pointer that gets passed to handlers. It is
actually implemented as a macro.
void
XML_UseParserAsHandlerArg(XML_Parser p)
After this is called, handlers receive the parser in the userData
argument. The userData information can still be obtained using the
XML_GetUserData function above.
int
XML_SetBase(XML_Parser p, const XML_Char *base)
Set the base to be used for resolving relative URIs in system identifiers.
The return value is 0 if there's no memory to store base, otherwise it's
non-zero.
const XML_Char *
XML_GetBase(XML_Parser p)
Return the base for resolving relative URIs.
int
XML_GetSpecifiedAttributeCount(XML_Parser p)
When attributes are reported to the start handler in the atts vector,
attributes that were explicitly set in the element occur before any attributes
that receive their value from default information in an ATTLIST declaration.
This function returns the number of attributes that were explicitly set, thus
giving the offset of the first attribute set due to defaults. It supplies
information for the last call to a start handler. If you're in a start handler,
then that means the current call.
int
XML_SetEncoding(XML_Parser p, const XML_Char *encoding)
Set the encoding to be used by the parser. It is equivalent to passing a
non-null encoding argument to the parser creation functions. It must not be
called after XML_Parser or XML_ParseBuffer have been called on the parser.
int
XML_SetParamEntityParsing(XML_Parser p, enum XML_ParamEntityParsing code)
If the parser wasn't compiled with the XML_DTD macro set, then this just
returns 0. Otherwise it returns 1 and enables parsing of parameter entities,
including the external parameter entity that is the external DTD subset,
according to code. The choices for code are:
