Microformats – building blocks for data-rich web pages

Microformats – building blocks for data-rich web pages
Latest microformats news
How to Consume Microformats 2 Data
A (very) belated follow up to
Getting Started with Microformats 2
, covering the basics of consuming and using microformats 2 data. Originally posted
on waterpigs.co.uk
More and more people are using microformats 2 to mark up profiles, posts, events and other data on their personal sites, enabling developers to build applications which use this data in useful and interesting ways. Whether you want to add basic support for webmention comments to your personal site, or have ambitious plans for a structured-data-aware-social-graph-search-engine-super-feed-reader, you’re going to need a solid grasp of how to parse and handle microformats 2 data.
Choose a Parser
To turn a web page containing data marked up with microformats 2 (or classic microformats, if supported) into a canonical MF2 JSON data structure, you’ll need a parser.
At the time of writing, there are actively supported
microformats 2 parsers
available for the following programming languages:
Go
Javascript (server-side and browser)
PHP
Python
Ruby
Rust
Parsers for various other languages exist, but might not be actively supported or support recent changes to the parsing specification.
There are also various websites which you can use to experiment with microformats markup without having to download a library and write any code:
My own live-updating
php-mf2 sandbox
The various parser comparison tools hosted on
microformats.io
Aaron Parecki
’s
pin13.net microformats parser
for parsing either URLs or HTML fragments
If there’s not currently a parser available for your language of choice, you have a few options:
Call the command-line tools provided by one of the existing libraries from your code, and consume the JSON they provide
Make use of one of the online mf2 parsers capable of parsing sites, and consume the JSON it returns (only recommended for very low volume usage!)
Write your own microformats 2 parser! There are plenty of people
happy to help
, and a language-agnostic test suite you can plug your implementation into for testing.
Considerations During Fetching and Parsing
Most real-world microformats data is fetched from a URL, which could potentially redirect to a different URL one or more times. The final URL in the redirect chain is called the “effective URL”. HTML often contains relative URLs, which need to be resolved against a base URL in order to be useful out of context.
If your parser has a function for “parsing microformats from a URL”, it should deal with all of this for you. If you’re making the request yourself (e.g. to use custom caching or network settings) and then passing the response HTML and base URL to the parser, make sure to
use the effective URL, not the starting URL!
The parser will handle relative URL resolution, but it needs to know the correct base URL.
When parsing microformats, an HTTP request which returns a non-200 value doesn’t necessarily mean that there’s nothing to parse! For example, a
410 Gone
response might contain a h-entry with a message explaining the deletion of whatever was there before.
Storing Raw HTML vs Parsed Canonical JSON vs Derived Data
When consuming microformats 2 data, you’ll most often be fetching raw HTML from a URL, parsing it to canonical JSON, then finally processing it into a simpler, cleaned and sanitised format ready for use in your website or application. That’s three different representations of the same data — you’ll most likely end up storing the derived data somewhere for quick access, but what about the other two?
Experience shows that, over time:
the way a particular application cleans up mf2 data will be tweaked and improved as you add new features and handle unexpected edge-cases
mf2 parsers gradually get improved, fixing bugs and occasionally adding entirely new features.
Therefore, if it makes sense for your use case, I recommend archiving a copy of the original HTML as well as your derived data, leaving out the intermediate canonical JSON. That way, you can easily create scripts or background jobs to update all the derived data based on the original HTML, taking advantage of both parser improvements and improvements to your own code at the same time, without having to re-fetch potentially hundreds of potentially broken links.
As mentioned in the previous section, if you archive original HTML for re-parsing, you’ll need to additionally store the effective URL for correct relative URL resolution.
For some languages, there are already libraries (such as
XRay
for PHP) which will perform common cleaning and sanitisation for you. If the assumptions with which these libraries are built suit your applications, you may be able to avoid a lot of the hard work of handling raw microformats 2 data structures!
If not, read on…
Navigating Microformat Structures
A parsed page may contain a number of microformat data structures (mf structs), in various different places.
Take a look at
the parsed canonical microformats JSON for the article you’re reading right now
, for example.
items
is a list of top-level mf structs, each of which may contain nested mf structs either under their
properties
or
children
keys.
Each individual mf struct is guaranteed to have at least two keys,
type
and
properties
type
is the primary way of identifying what sort of thing that struct represents (e.g. a person, a post, an event). Structs can have more than one type if they represent multiple things at once without wanting to nest them — for example, a post detailing an event might be both a h-entry and a h-event at the same time. Structs can also have additional top-level keys such as
id
and
lang
Generally speaking,
type
information is most useful when dealing with top-level mf structs, and mf structs nested under a
children
key. Nested mf structs found in
properties
will also have
type
information, but their usage is usually implied by the property name they’re found under.
For many common use cases (e.g. a homepage feed and profile) there are several different ways people might nest mf structs to achieve the same goals, so it’s important that your code is capable of searching the entire tree, rather than just looking at the top-level mf structs.
Never assume that the microformat struct you’re looking for will be in the top-level of the
items
list!
You need to search the whole tree.
I recommend writing some functions which can traverse a mf tree and return all structs which match a filtering callback. This can then be used as a basis for writing more specific convenience functions for common tasks such as finding all microformats on a page of a particular type, or where a certain property matches a certain value.
See
my microformats2 PHP functions
for some working examples.
Possible Property Values
Each key in a mf struct’s
properties
dict maps to a list of values for that property. Every property may map to multiple values, and those values may be a mixture of any of the following:
A plain string value, containing no HTML, and leaving HTML entities unescaped (e.g.
"items"
: [{
"type"
: [
"h-card"
],
"properties"
: {
"name"
: [
"Barnaby Walters"
}]
(In future examples I will leave out the encapsulating
{"items": [{"type": [•••], •••}]}
for brevity, focusing on the
properties
key of a single mf struct.)
An embedded HTML struct, containing two keys:
html
, which maps to an HTML representation of the property, and
value
, mapping to a plain text version.
"properties": {
"content": [{
"html": "

The content
of
a post,
as
raw HTML (
or
not
).

",
"value": "The content
of
a post,
as
raw HTML (
or
not
)."
}]
An img/alt struct, containing the URL of a parsed image under
value
, and its alt text under
alt
"properties"
: {
"photo"
: [{
"value"
"https://example.com/profile-photo.jpg"
"alt"
"Example Person"
}]
A nested microformat data structure, with an additional
value
key containing a plaintext representation of the data contained within.
"properties"
: {
"author"
: [{
"type"
: [
"h-card"
],
"properties"
: {
"name"
: [
"Barnaby Walters"
},
"value"
"Barnaby Walters
}]
All properties may have more than one value. In cases where you expect a single property value (e.g.
name
), simply take the first one you find, and in cases where you expect multiple values, use all values you consider valid. There are also some cases where it may make sense to use multiple values, but to prioritise one based on some heuristic — for example, an h-card may have multiple
url
values, in which case the first one is usually the “canonical” URL, and further URLs refer to external profiles.
Let’s look at the implications of each of the potential property value structures in turn.
Firstly,
Never assume that a property value will be a plaintext string
. Microformats publishers can nest microformats, embedded content and img/alt structures in a variety of different ways, and your consuming code should be as flexible as possible.
To partially make up for this complexity, you can
always rely on the
value
key of nested structs to provide you with an equivalent plaintext value
, regardless of what type of struct you’ve found.
When you start consuming microformats 2, write a function like this, and get into the habit of using it
every time
you want a single, plaintext value from a property:
def
get_first_plaintext
(mf_struct, property_name)
try
first_val = mf_struct[
'properties'
][property_name][
if
isinstance(first_val, str):
return
first_val
else
return
first_val[
'value'
except
(IndexError, KeyError):
return
None
Secondly,
Never assume that a particular property will contain an embedded HTML struct
— this usually applies to
content
, but is relevant anywhere your application expects embedded HTML. If you want to reliably get a value encoded as raw HTML, then you need to:
Check whether the first property value is an embedded HTML struct (i.e. has an
html
key). If so, take the value of the
html
key
Otherwise, get the first plaintext property value using the approach above, and HTML-escape it
If neither is found, the property has no value.
In Python 3.5+, that could look something like this:
from
html
import
escape
def
get_first_html
(mf_struct, property_name)
try
first_val = mf_struct[
'properties'
][property_name][
if
isinstance(first_val, dict)
and
'html'
in
first_val:
return
first_val[
'html'
else
plaintext_val = get_first_plaintext(mf_struct, property_name)
if
plaintext_val
is
not
None
plaintext_val = escape(plaintext_val)
return
plaintext_val
except
(IndexError, KeyError):
return
None
In some cases, it may make sense for your application to be aware of whether a value was parsed as embedded HTML or a plain text string, and to store/treat them differently. In all other cases,
always
use a function like this when you’re expecting embedded HTML data.
Thirdly, when expecting an image URL, check for an img/alt structure, falling back to the plain text value (and either assuming an empty alt text or inferring an appropriate one, depending on your specific use case). Something like this could be a good starting point:
def
get_img_alt
(mf_struct, property_name)
try
first_val = mf_struct[
'properties'
][property_name][
if
isinstance(first_val, dict)
and
'alt'
in
first_val:
return
first_val
else
plaintext_val = get_first_plaintext(mf_struct, property_name)
if
plaintext_val
is
not
None
return
'value'
: plaintext_val,
'alt'
''
return
None
except
(IndexError, KeyError):
return
None
Finally, in cases where you expect a nested microformat, you might end up getting something else. This is the hardest case to deal with, and the one which depends the most on the specific data and use-case you’re dealing with. For example, if you’re expecting a nested h-card under an
author
property, but get something else, you could use any of the following approaches:
If you got a plain string which doesn’t look like a URL, treat it as the
name
property of an implied h-card structure with no other properties (and if you need a URL, you could potentially take the hostname of the effective URL, if it works in context as a useful fallback value)
If you got an img alt struct, you could treat the
value
as the
photo
property, the
alt
as the
name
property, and potentially even take the hostname of the
photo
URL to be the implied fallback
url
property (although that’s pushing it a bit, and in most cases it’s probably better to just leave out the
url
If you got an embedded HTML struct, take its plaintext
value
and use one of the first two approaches
If you got a plain string, check to see if it looks like a URL. If so, fetch that URL and look for a representative h-card to use as the author value
If you get an embedded mf struct with a
url
property but no
photo
, you could fetch the
url
, look for a representative h-card (more on that in the next section) and see if it has a
photo
property
Treat the
author
property as invalid and run the h-entry (or entire page if relevant) through the
authorship algorithm
The first three are general principles which can be applied to many scenarios where you expect an embedded mf struct but find something else. The last three, however, are examples of a common trend in consuming microformats 2 data: for many common use-cases, there are well-thought-through algorithms you can use to interpret data in a standardised way.
Know Your Algorithms and Vocabularies
The authorship algorithm mentioned above is one of several more-or-less formally established algorithms used to solve common problems in indieweb usages of microformats 2. Some others which are worth knowing about include:
“Who wrote this post?”:
authorship algorithm
“There’s more than one h-card on this page, which one should I use?”:
representative h-card
“I want to get a paginated feed of posts from this page”:
How to consume h-feed
“How do I find and display the main post on this page?”:
How to consume h-entry
“I received a response to one of my posts via webmention, how do I display it?”:
How to display comments
Library implementations of these algorithms exist for some languages, although they often deviate slightly from the exact text. See if you can find one which meets your needs, and if not, write your own and share it with the community!
In addition to the formal consumption algorithms, it’s worth looking through the definitions of the microformats vocabularies you’re using (as well as testing with real-world data) and adding support for properties or publishing techniques you might not have thought of the first time around. Some examples to get you started:
If an h-card has no valid
photo
, see if there’s a valid
logo
you can use instead
When presenting a h-entry with a featured photo, check both the
photo
property and the
featured
property, as one or the other might be used in different scenarios
When dealing with address or location data (e.g. on an h-card, h-entry or h-event), be aware that either might be present in various different forms. Co-ordinates might be separate
latitude
and
longitude
properties, a combined plaintext
geo
property, or an embedded
h-geo
. Addresses might be separate top-level properties or an embedded h-adr. There are many variations which are totally valid to publish, and your consuming code should be as liberal as possible in what it accepts.
If a h-entry contains images which are marked up with
u-photo
within the
e-content
, they’ll be present both in the
content
html
key and also under the
photo
property. If your app shows the embedded
content
HTML rather than using the plaintext version, and also supports
photo
properties (which may also be present outside the
content
), you may have to sniff the presence of photos within the
content
, and either remove them from it or ignore the corresponding
photo
properties to avoid showing photos twice.
Sanitise, Validate, and Truncate
In the vast majority of cases, consuming microformats 2 data involves handling, storing and potentially re-publishing untrusted and potentially dangerous input data. Preventing XSS and other attacks is out of the scope of the microformats parsing algorithm, so the data your parser gives you is just as dangerous as the original source. You need to take your own measures for sanitising and truncating it so you can store and display it safely.
Covering every possible injection and XSS attack is out of the scope of this article, so I highly recommend referring to the OWASP resources on
XSS Prevention
Unicode Attacks
and
Injection Attacks
for more information.
Other than that, the following ideas are a good start:
Use plaintext values where possible, only using embedded HTML when absolutely necessary
Pass everything (HTML or not) through a well-respected HTML sanitizer such as PHP’s
HTML Purifier
. Configure it to make sure that embedded HTML can’t interfere with your own markup or CSS. It probably shouldn’t contain any javascript ever, either.
In any case where you’re expecting a value with a specific format, validate it as appropriate.
More specifically, everywhere that you expect a URL, check that what you got was actually a URL. If you’re using the URL as an image, consider fetching it an checking its content type
Consider either proxying resource such as images, or storing local copies of them (reducing size and resolution as necessary), to avoid mixed content issues, potential attacks, and missing images if the links break in the future.
Decide on relevant maximum length values for each separate piece of external content, and truncate them as necessary. Ideally, use a language-aware truncation algorithm to avoid breaking words apart. When the content of a post is truncated, consider adding a “Read More” link for convenience.
Test with Real-World Data
The web is a diverse place, and microformats are a flexible, permissive method of marking up structured data. There are often several different yet perfectly valid ways to achieve the same goal, and as a good consumer of mf2 data, your application should strive to accept as many of them as possible!
The best way to test this is with
real world data
. If your application is built with a particular source of data in mind, then start off with testing it against that. If you want to be able to handle a wider variety of sources, the best way is to determine what vocabularies and publishing use-cases your application consumes, and look at the Examples sections of the relevant
indieweb.org
wiki pages for real-world sites to test your code against.
Don’t forget to test your code against examples you’ve published on your own personal site!
Next Steps
Hopefully this article helped you avoid a lot of common gotchas, and gave you a good head-start towards successfully consuming real-world microformats 2 data.
If you have questions or issues, or want to share something cool you’ve built, come and join us in the
indieweb chat room
February 19th, 2022
waterpigs.co.uk/
Comments Off
on How to Consume Microformats 2 Data
Google confirms Microformats are still a recommended metadata format for content
This post
originally appeared on Jamie Tanna’s site
Google announced that they are
removing support for the data-vocabulary metadata
markup that could be used to provide rich search results on its Search Engine.
In a Twitter exchange, John Mueller, a Webmaster Trends Analyst at Google, confirmed that
Microformats
are still being supported by Google at this time:
Yes, we still support them.
— 🍌 John 🍌 (@JohnMu)
January 21, 2020
John also confirmed that he knows of no upcoming plans to deprecate Microformats:
We don’t have any plans for changes to announce there at the moment. I don’t know off-hand how broadly microformats are used, my guess is it’s much more than data-vocabulary. That said …
— 🍌 John 🍌 (@JohnMu)
January 21, 2020
This is an especially great result due to the way that Google is quite happy to abandon various metadata formats, as noted in our
7th anniversary blog post
, almost 8 years ago. With this announcement, Microformats are now the longest-supported metadata format that Google parses,
since at least 2009
With the continued growth of Microformats across the
IndieWeb
, we expect that Google will extend its Microformats support accordingly.
Tags for this entry
indieweb
microformats2
March 4th, 2020
jamietanna
1 Comment
microformats.org Year 14 — Welcome New Admins
In microformats.org year 14, we welcome
new admins
Aaron Parecki
Gregor Morrill
Martijn van der Ven
, and
Sven Knebel
! All have been active for years, helping welcome new members and doing essential wiki gardening &
microformats2 parser updates
Originally posted at:
tantek.com
Tags for this entry
microformats2
June 22nd, 2018
Tantek
Comments Off
on microformats.org Year 14 — Welcome New Admins
Happy 13th to microformats.org!
With more use of
microformats2
, especially among the growing
indieweb
network of websites, we’ve iterated
key
specs
for real-world needs and are seeing more active community members. More updates & posts coming up!
Originally posted on
tantek.com
Tags for this entry
indieweb
microformats2
June 21st, 2018
Tantek
Comments Off
on Happy 13th to microformats.org!
Improving the php-mf2 parser
During the past year, the popular
php-mf2
microformats parser has received quite a few improvements. My site runs ProcessWire and one of the plugins for it uses php-mf2, so I have been spending some time on it.
My own experience with microformats started when I discovered the
hCard microformat
. I was impressed with the novelty of adding some simple HTML classes around contact information and having a browser extension parse it into an address book. Years later, when I started to get involved in the IndieWeb community, I learned a lot more about microformats2 and they became a key building block of my personal site.
php-mf2 is now much better at backwards-compatible parsing of microformats1. This is important because software should be able to consistently consume content whether it’s marked up with microformats1, microformats2, or a combination. An experimental feature for parsing language attributes has also been added. Finally, it’s now using the microformats test suite. Several other parsers use this test suite as well. This will make it easier to catch bugs and improve all of the different parsers.
php-mf2 is a stable library that’s ready to be installed in your software to start consuming microformats. It is currently used in
Known
WordPress plugins
, and
ProcessWire plugins
for richer social interactions. It’s also used in tools like
XRay
and
microformats.io
. I’m looking forward to more improvements to php-mf2 in the coming year as well as more software using it!
Original published at:
June 22nd, 2017
gRegor Morrill
Comments Off
on Improving the php-mf2 parser
Browse all entries by month in the
blog archive
What are microformats?
Designed for humans first and machines second, microformats are a set of simple, open data formats built upon existing and widely adopted standards.
Learn more about microformats
Microformat specifications
People and Organizations
h-card
XFN
Calendars and Events
h-calendar
Opinions, Ratings and Reviews
h-review
Licenses:
rel-license
Tags, Keywords, Categories
rel-tag
Lists and Outlines
XOXO
More…
See
the list of all microformats
Upcoming Events
See microformats events on the wiki
See also
IndieWebCamp Events!
Post Categories
Events
News
This Week in Microformats
WordPress
| Hosting sponsored by
Linode
No WWW