In IIS, you could define custom error handlers using the IIS admin tool.  In Azure, you don't have access to the IIS admin tool, so you have to find new ways of doing what you used to do with that tool.  The secret is to learn the IIS 7 integrated pipeline.  IIS 7 has everything in configuration, and you can accomplish many of the configuration tasks simply in your web.config.

The best recommendation is to use Windows Vista for your development machine (if you don't already do this) or to publish to a Windows 2008 Server, both of which are equiped with IIS 7.

For the error handler problem, you will find that you can configure error handling in the system.webServer section.  For example, if you want to direct all 404 errors to /page-not-found.aspx, simply include the following in your web.config file:

<system.webServer>
    <httpErrors errorMode="Custom">
        <remove statusCode="404" />
        <error statusCode="404" responseMode="ExecuteURL" path="/page-not-found.aspx"/>
    </httpErrors>
</system.webServer>

If you are hosting your application under Windows Vista (not just under the dev fabric), you may get an error with this configuration because the default IIS installation does not allow you to override the httpErrors section.  Go to your c:\windows\system32\inetsrv\config\applicationHost.config file, search for "httpErrors" and change the overrideModeDefault attribute from Deny to Allow.