By: Joc Triglav

In 1993, while I was teaching at the British Columbia Institute of Technology and Dale was still at MDA, we started to discuss a significant problem that we had both noticed in the GIS market; moving spatial data between different systems was complex, time-consuming, and in many cases, impossible. At first, we thought that there must be someone out there who had already built a solution to address such a common problem. But after further research, it turned out that although this was a significant issue in the market, there really wasn’t any software solution available to address it. So, we decided to start a business (Safe Software) that focuses on solving this challenge. Our initial intent was to support the Spatial Archive and Interchange Format (SAIF), pronounced ‘safe’, a format developed by the BC Government Mapping Agency. As you can probably guess, this is also where we got our company name. Together we built FME, a spatial ETL platform that enables organizations to translate, transform, federate and distribute spatial data in over 200 different formats. Our focus over the past decade has been on continuing to add support for the plethora of new data formats – whether it’s a new raster format, an emerging 3D or BIM format, or a new web format. The thing that has really driven us from the start is making our customers happy. We still believe that if we look after our clients everything else will look after itself. We often say that Safe is like a restaurant; to be successful you need to have both good products and good service. The only difference really is that we are serving software rather than food. Every GIS professional or user faces data interoperability headaches. But, the spectrum of challenges related to data interoperability is very diverse. How does your spatial ETL technology help overcome such a diverse set of problems?

Dale: At one point or another, most organizations need to acquire spatial data, store it, format it or share it. Our spatial ETL technology, FME, offers four key capabilities to help power the flow of spatial data across this GIS process. The first capability is data translation, that is, helping organizations read or write data from one format to another. As Don mentioned earlier, FME supports data translation in hundreds of GIS, CAD, raster and database formats – with new formats being added all the time. At first it’s a little hard to believe, but it’s true – the world continues to constantly create new formats!

Don: Last year we added support for raster formats. While adding support for raster data is exciting, the real excitement is now users can have a single spatial ETL workflow that integrates both raster and vector so they can perform interesting transformations across multiple datasets. This integration, fusion, mashups, or whatever you want to call it, is very exciting!

Dale: Aside from reading and writing data in many formats, the real power of our technology is what you can do with the data when you are moving it from the source to the destination system. This is the second capability, data transformation, and it’s really the key to the power of true spatial ETL. Through a drag-and-drop interface (Workbench), FME enables users to restructure data as it moves from a source system to the destination system. They can use it to restructure the data schema for one format or between multiple formats.

Don: Many customers also use our Universal Viewer as a way to perform quality assurance ‘inspections’ on the data throughout the spatial ETL process. The first step for success in spatial ETL is to first ensure that you truly understand your data!

Dale: Data federation is the third capability required for ‘true’ spatial ETL. It allows you to create a unified view of spatial data from multiple sources but without having to move the data into a single location. This ‘on-the-fly’ process makes it very efficient for orga_ni_zations to quickly present different data views to different user communities, without taxing IT bandwidth or resources.

Don: We’ve traditionally provided data integration capabilities and are expanding this to support data federation. With FME, users simply define the mapping from the source data to the ‘federated view’ of data. Using this approach it is easy to bring new sources online and not worry that they are going to negatively impact other data sources.

Dale: With the world starting to take greater interest in location-based or spatial data, we’ve invested in expanding FME’s distribution capabilities, the final element required to power flow of spatial data across the GIS process. We’re introducing a new solution this year (FME Server) that focuses on spatial data distribution. FME Server will offer a variety of web services that will make it very efficient for organizations to make data available in different users’ requested formats and data models.

A GeoRSS feed provided by the USGS where each histogram represents the location of a recent earthquake,
with magnitude indicated by the height of the histogram. This GeoRSS feed was transformed into a KML file using FME,
then visualized in Google Earth.

Don: The first data manipulation problem that we focused on in building FME was the CAD <-> GIS problem. This is still one of the most common things that people use FME for. One fundamental difference between these two types of systems is that CAD systems often don’t represent polygons directly, but they instead store the lines that make up the polygons. There is also typically a point in the middle of the ‘polygons’ which has a link to an attribute table in a database of flat file. To move data from CAD to GIS, people use FME to make the polygons from the line work, then match the point to the interior of the polygons and finally to join the attributes from the table to the polygon so that we have a nice polygonal GIS dataset. FME can also support the opposite workflow, that is, enabling users to bring GIS data into their CAD system. In general, FME enables users to efficiently extract the intent of the CAD data and make it explicit in the destination system. For example, taking the name of a symbol at a certain location and applying a rule that transforms it into a water value of a certain capacity and installation date.

Dale: Of course, something that users often discover when using FME is data quality issues. They may have lines that cross or don’t meet or polygons that overlap that shouldn’t. Data quality assurance is another task that people often use FME for.

Don: FME was architected for flexibility. FME doesn’t automatically fix your data, but instead, FME presents users with options so they can decide what they want to do. There is no way, for example, that anyone would want software to magically ‘fix’ legal boundaries just because it found out that the geometric features don’t line up the way the user thought they did.

Dale: Regardless of what you want to use FME for, the most important thing that users must do first is really understand the data that they want to move. To do this easily, we’ve included a data inspection tool in FME. Using this tool, users can really take a good look at their data in any of the supported formats. This is a unique capability of our product. We’re not aware of any other viewer in the market that can do this.

3D image of Schutterstock.

Dale: The big story with FME 2007 was the ‘official release’ of our raster support. With this release, users can not only process vector data, but also imagery and measurement data such as Digital Elevation Models (DEM). While this is exciting, the true power of FME 2007 is that users can now perform spatial ETL operations that combine or fuse data from both raster and vector.

Don: Continuing with the ‘fusion’ theme, FME 2007 also introduced integration with Informatica PowerCenter and Microsoft SQL Server Integration Services. For the first time, organizations now have a way to make their mapping data available to their corporate IT infrastructure and to easily bring data from the IT world into the mapping department. With the 2007 release, FME truly enables organizations to build the infrastructure needed for moving their spatial data.

The diagram shows how FME supports various stages in the ETL process.

Don: FME Server is a powerful and scalable spatial ETL processing environment that will enable organizations to very efficiently translate, transform, federate and distribute spatial data to a variety of user communities in the structure and format they desire – and via the web technology they prefer. Basically, it allows organizations to exploit all of the power of FME Desktop in a server environment plus more.

Dale: In terms of scalability, FME Server is designed in a way that allows organizations to add multiple FME engines to their architecture. More FME engines essentially result in more scalable spatial ETL processing.

Don: FME Server is based on the evolution of the architecture from our SpatialDirect product (SpatialDirect is our first generation web-based solution for data delivery.) FME Server can be leveraged by organizations in four ways.

a) Simple Data Processing: For current FME Desktop customers, any existing spatial ETL workflow ("workspace") can be installed on FME Server. Once this is done any user in the organization – not just GIS-savvy users - can then run that workspace using a simple web interface. For example, you could upload a workspace that converts any GeoTiff to any other raster file format. Once uploaded, any user can then run it – without having any background in GIS. Of course, only users with the authoring environment (provided by the FME Desktop) can create new workspaces.

b) Web Services: FME Server can enable organizations to efficiently provide spatial data download services to diverse user communities. Our goal is to enable FME Server to provide data to any web service protocols that become available. We are essentially treating web services on the server just as we did with format support on the desktop.

c) Powerful Spatial ETL Processing: Some organizations have a large number of spatial ETL tasks that run daily or weekly. With FME Server, these organizations will have access to a scalable ‘enterprise-ready’ spatial ETL processing engine to enable faster, more efficient processing of these jobs.

d) Integration with Traditional ETL Systems: We are seeing more and more organizations that want to move data across the organization. In the past, the mapping department has been ‘special’ in that its data was not to be disseminated freely throughout the organization or combined with other departments’ data. This is changing as organizations are starting to realize the power of combining spatial and non-spatial datastores together to drive more effective decision-making. To support this, FME Server offers application extensions for integration with tools such as Informatica PowerCenter or IBM Information Server Data Stage. FME Server essentially acts as a bridge between these systems and spatial data sources to enable organizations to easily move data between the mapping department and the rest of the organization.

Dale: As you can see, FME Server will be able to meet a variety of common problems in data interoperability. It’s available in beta now and is due for final release in the first half of 2008.

Example of a raster image, a key feature in FME 2007.

How does FME fit into the GIS industry in terms of the other types of technology and software available?

Don: FME is focused on solving spatial ETL challenges. Spatial ETL plays a vital role in the GIS industry overall as it helps data flow seamlessly throughout the GIS process, while protecting the integrity of the data structure as it is translated and transformed from one format or location to another. Our focus is on supporting the data interoperability needs of the other vendors in the GIS industry. In fact, most of the leading GIS vendors have partnered with us to integrate core FME capabilities right within their own applications so they can better address their customers’ needs for true data interoperability. For example, FME is available to ESRI ArcGIS users through a Data Interoperability Extension. FME can also exploit the spatial and attribute SQL queries offered by supported spatial databases including ESRI ArcSDE, Oracle Spatial, PostGIS and others.

Emerging open standard formats and web services offer new approaches to addressing the problems of data usage and integration in the geospatial context. How does FME address both these new capabilities while still meeting the needs for traditional spatial data formats?

Don: Raster was the major theme for our FME 2007 release. For example, users can now load data into ESRI ArcSDE or Oracle GeoRaster. For FME 2008, we are being very ambitious and there are actually two focus areas. The first focus is in adding support for 3D data formats. For example, building information model (BIM) formats such as IFC. The second focus for FME 2008 is on bringing spatial ETL to the web. This is not just about FME Server providing services to the web through a Service Oriented Architecture (SOA). It’s also about enabling our core FME engine to consume services so that the FME users are able to exploit them.

Dale: FME can essentially address both the need to provide spatial data – through FME Server – and to consume spatial data – through FME Desktop. If you download the latest beta of FME Desktop and launch the Workbench component, you will find a new transformer category called ‘Web Services’. This category includes several transformers that are focused purely on enabling users to easily access web pages and web services from within a workspace. This is exciting as it essentially turns our FME platform into a web service client so that the power of its transformation engine can be leveraged on the output of the new data sources and services that are being developed. This fits nicely with our vision for FME in that it will allow organizations to take a simpler approach to delivering spatial data to a destination system or user.

Don: Several of our readers in FME can also read data directly from the web; for example, GeoRSS/RSS, GeoJson/Json and WMS/WFS. The web is a whole new platform for our techno_logy. By making it easy for organizations to create and consume services, more and more users will be exposed to the benefits of our technology.

There is a saying ‘The good thing about standards is that there are so many to choose from’. In your opinion, what are the arguments – if any – in favour of the expectation that this "Babel" situation of file and data formats is ever going to change?

Dale: The thing about technology is that it is difficult to predict where things are going to come from. For example, three years ago, who would have thought that Google and Microsoft would be driving forces in web mapping? There are a lot of standards out there for data interoperability. Our approach is to support all standards - both official and de facto.

Don: For the web, we are focused on supporting all the different standards, just as we have viewed format and database support on the desktop. For example, FME was one of the first software solutions to add support for GeoRSS and GeoJSON.

Dale: Right now in the web world in general and the web mapping world in particular, things are changing at a tremendous rate. This makes it very difficult for standards to be established as users are still figuring out all the things that they want done and new technology is being released at a seemingly increasing rate. There are, however, a few things that are going to remain important for the foreseeable future. For example, many organizations are building new formats around XML and GML.

Don: KML is also something that we are seeing more and more use of. OGC ratification of a future version of KML will certainly solidify its importance for a range of important purposes.

Dale: Right now every day is an exciting day as the pace of change and opportunity continue to grow. Safe’s niche in the market is a really fun place to be as applications are always going to need access to data and we’re all about making data available to the masses no matter where that data originates.

How do the latest geospatial ‘mass-players’ like Google, Microsoft and others affect your business and technology strategies?

Don: These vendors have changed everything. Their influence in the market has led to a significant increase in the demand for spatial data and the level of user expectations. As our company focuses on enabling applications to get the data they need, this change has been very good for Safe. To us, these new tools are simply new data endpoints that we need to support - very popular endpoints, with lots of users. For example, we will support Microsoft both on the web and at the database level.

Dale: Over the last two years the number of web based technologies that have arrived on the scene is very exciting. We now have GeoRSS GeoJSON, Yahoo Pipes, and countless new web services coming online all the time. We’re pretty excited about the upcoming launch of FME 2008 as it will be able to address this both from the server and the client level by giving users building blocks to take advantage of web services and to easily create their own web services.

Don: Take GeoRSS for example. One can easily publish spatial data to the web by simply writing a GeoRSS file and then copy to their website. That is it, and then users of Google Maps or Virtual Earth can easily see on a zoomable, ‘slippy’ map. It’s amazing!

Not only is your company known for its quality support service and dynamic online community, it’s also known for its candid openness about future plans. Can you give us a sneak preview of what’s next?

Dale: As you can tell, we’re really excited about our upcoming FME 2008 release which will add support for 3D data formats and introduce FME Server. Our focus this year is to continue to deliver new and better ways for removing data barriers to applications. Data is to applications as fuel is to an engine. With FME in 2008, we are going to bring very high test fuel to both spatial and non-spatial applications.

Joc Triglav  is editor and columnist of GeoInformatics. For more information on the subject visit www.safe.com.